[jira] [Commented] (HDFS-12955) [SPS]: Move SPS classes to a separate package
[ https://issues.apache.org/jira/browse/HDFS-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301064#comment-16301064 ] Uma Maheswara Rao G commented on HDFS-12955: +1 I have just pushed to the branch. Thanks Rakesh, this helps to start refactoring. > [SPS]: Move SPS classes to a separate package > - > > Key: HDFS-12955 > URL: https://issues.apache.org/jira/browse/HDFS-12955 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nn >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R >Priority: Trivial > Attachments: HDFS-12955-HDFS-10285-00.patch, > HDFS-12955-HDFS-10285-01.patch > > > For clean modularization, it would be good if we moved SPS related classes > into its own package. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12955) [SPS]: Move SPS classes to a separate package
[ https://issues.apache.org/jira/browse/HDFS-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301041#comment-16301041 ] Surendra Singh Lilhore commented on HDFS-12955: --- Thanks [~rakeshr] for clarification. bq. I have named the package namenode.sps, keeping in mind that the current implementation classes are specific to the NN side service. Does this make sense to you? ok.. > [SPS]: Move SPS classes to a separate package > - > > Key: HDFS-12955 > URL: https://issues.apache.org/jira/browse/HDFS-12955 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nn >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R >Priority: Trivial > Attachments: HDFS-12955-HDFS-10285-00.patch, > HDFS-12955-HDFS-10285-01.patch > > > For clean modularization, it would be good if we moved SPS related classes > into its own package. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12955) [SPS]: Move SPS classes to a separate package
[
https://issues.apache.org/jira/browse/HDFS-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301029#comment-16301029
]
Rakesh R commented on HDFS-12955:
-
Thanks [~surendrasingh] for the reviews.
bq. I feel sps package name should be org.apache.hadoop.hdfs.server.sps
I have named the package {{namenode.sps}}, keeping in mind that the current
implementation classes are specific to the NN side service. Does this make
sense to you?
> [SPS]: Move SPS classes to a separate package
> -
>
> Key: HDFS-12955
> URL: https://issues.apache.org/jira/browse/HDFS-12955
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nn
>Affects Versions: HDFS-10285
>Reporter: Uma Maheswara Rao G
>Assignee: Rakesh R
>Priority: Trivial
> Attachments: HDFS-12955-HDFS-10285-00.patch,
> HDFS-12955-HDFS-10285-01.patch
>
>
> For clean modularization, it would be good if we moved SPS related classes
> into its own package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12955) [SPS]: Move SPS classes to a separate package
[
https://issues.apache.org/jira/browse/HDFS-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300994#comment-16300994
]
genericqa commented on HDFS-12955:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
11s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-10285 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 24s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestErasureCodingMultipleRacks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12955 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903344/HDFS-12955-HDFS-10285-01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a9b16116a12d 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-10285 / 73f7db6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22496/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22496/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdf
[jira] [Commented] (HDFS-12860) StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300987#comment-16300987
]
SammiChen commented on HDFS-12860:
--
Hi [~eddyxu], thanks for report the issue and work on it.
1. It's great to add error message to provide more information when
Precondition check fails. There are "%d" used in String.format and "%s" used
in Preconditions. Is it because Preconditions doesn't support "%s"?
2. ")" is missed in {{AlignedStripe.toString}} and {{StripingCell.toString}}
3. Can you add some javadoc or comment in
{{testDivideOneStripeLargeBlockSize}}? If we want to test block group larger
than 2GB, use the RS-6-3-1024k as an example, the {{stripSize}} is 9 * 1MB,
{{stripesPerBlock}} will be > (2 * 1024) / 9M, {{blockSize}} is {{cellSize *
stripesPerBlock}}.Also I would suggest add a end-to-end test case in
{{TestErasureCodingPolicies}}.
> StripedBlockUtil#getRangesInternalBlocks throws exception for the block group
> size larger than 2GB
> --
>
> Key: HDFS-12860
> URL: https://issues.apache.org/jira/browse/HDFS-12860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12860.00.patch
>
>
> Running terasort on a cluster with 8 datanodes, 256GB data, using
> RS-3-2-1024k.
> The test data was generated by {{teragen}} with 32 mappers.
> The terasort benchmark fails with the following stack trace:
> {code}
> 17/11/27 14:44:31 INFO mapreduce.Job: map 45% reduce 0%
> 17/11/27 14:44:33 INFO mapreduce.Job: Task Id :
> attempt_1510080297865_0160_m_08_0, Status : FAILED
> Error: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil$VerticalRange.(StripedBlockUtil.java:701)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getRangesForInternalBlocks(StripedBlockUtil.java:442)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.divideOneStripe(StripedBlockUtil.java:311)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:308)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:562)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12955) [SPS]: Move SPS classes to a separate package
[
https://issues.apache.org/jira/browse/HDFS-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300984#comment-16300984
]
Surendra Singh Lilhore commented on HDFS-12955:
---
Thanks [~rakeshr] for patch.
I feel sps package name should be {{org.apache.hadoop.hdfs.server.sps}}. We are
planning to start SPS as a separate service, so it should follow other service
package name
{noformat}
org.apache.hadoop.hdfs.server.datanode
org.apache.hadoop.hdfs.server.namenode
org.apache.hadoop.hdfs.server.mover
{noformat}
> [SPS]: Move SPS classes to a separate package
> -
>
> Key: HDFS-12955
> URL: https://issues.apache.org/jira/browse/HDFS-12955
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nn
>Affects Versions: HDFS-10285
>Reporter: Uma Maheswara Rao G
>Assignee: Rakesh R
>Priority: Trivial
> Attachments: HDFS-12955-HDFS-10285-00.patch,
> HDFS-12955-HDFS-10285-01.patch
>
>
> For clean modularization, it would be good if we moved SPS related classes
> into its own package.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-12935: - Affects Version/s: 3.0.0 Status: Patch Available (was: In Progress) HDFS-12935.002.patch： Patch to fix ten commands mentioned above, and also add unit test cases for all of them. > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0, 3.0.0-beta1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang > Attachments: HDFS-12935.002.patch, HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12947) Limit the number of Snapshots allowed to be created for a Snapshottable Directory
[ https://issues.apache.org/jira/browse/HDFS-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300959#comment-16300959 ] Xiao Chen commented on HDFS-12947: -- Idea of adding a config sounds good, but -1 on changing the default for compat reasons. > Limit the number of Snapshots allowed to be created for a Snapshottable > Directory > - > > Key: HDFS-12947 > URL: https://issues.apache.org/jira/browse/HDFS-12947 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12947.001.patch > > > Currently, A snapshottable directory is able to accommodate 65,536 snapshots. > In case a directory has large no of snapshots , deletion of any of the > earlier snapshots take a lot of time which might lead to namenode crash > (HDFS-11225). > This jira is introduced to limit the no of the snapshots under a > snapshottable directory to a reasonable value(say 10) which can be overriden. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300947#comment-16300947
]
Xiao Chen commented on HDFS-12528:
--
No problem, I'll take a crack soon. Thanks John!
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12528) Short-circuit reads unnecessarily disabled for a long time
[
https://issues.apache.org/jira/browse/HDFS-12528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen reassigned HDFS-12528:
Assignee: Xiao Chen
> Short-circuit reads unnecessarily disabled for a long time
> --
>
> Key: HDFS-12528
> URL: https://issues.apache.org/jira/browse/HDFS-12528
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, performance
>Affects Versions: 2.6.0
>Reporter: Andre Araujo
>Assignee: Xiao Chen
> Attachments: HDFS-12528.000.patch
>
>
> We have scenarios where data ingestion makes use of the -appendToFile
> operation to add new data to existing HDFS files. In these situations, we're
> frequently running into the problem described below.
> We're using Impala to query the HDFS data with short-circuit reads (SCR)
> enabled. After each file read, Impala "unbuffer"'s the HDFS file to reduce
> the memory footprint. In some cases, though, Impala still keeps the HDFS file
> handle open for reuse.
> The "unbuffer" call, however, causes the file's current block reader to be
> closed, which makes the associated ShortCircuitReplica evictable from the
> ShortCircuitCache. When the cluster is under load, this means that the
> ShortCircuitReplica can be purged off the cache pretty fast, which closes the
> file descriptor to the underlying storage file.
> That means that when Impala re-reads the file it has to re-open the storage
> files associated with the ShortCircuitReplica's that were evicted from the
> cache. If there were no appends to those blocks, the re-open will succeed
> without problems. If one block was appended since the ShortCircuitReplica was
> created, the re-open will fail with the following error:
> {code}
> Meta file for BP-810388474-172.31.113.69-1499543341726:blk_1074012183_273087
> not found
> {code}
> This error is handled as an "unknown response" by the BlockReaderFactory [1],
> which disables short-circuit reads for 10 minutes [2] for the client.
> These 10 minutes without SCR can have a big performance impact for the client
> operations. In this particular case ("Meta file not found") it would suffice
> to return null without disabling SCR. This particular block read would fall
> back to the normal, non-short-circuited, path and other SCR requests would
> continue to work as expected.
> It might also be interesting to be able to control how long SCR is disabled
> for in the "unknown response" case. 10 minutes seems a bit to long and not
> being able to change that is a problem.
> [1]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/BlockReaderFactory.java#L646
> [2]
> https://github.com/apache/hadoop/blob/f67237cbe7bc48a1b9088e990800b37529f1db2a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/shortcircuit/DomainSocketFactory.java#L97
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12955) [SPS]: Move SPS classes to a separate package
[ https://issues.apache.org/jira/browse/HDFS-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-12955: Attachment: HDFS-12955-HDFS-10285-01.patch Attached another patch fixing checkstyle warnings. Findbug warning & test case failures are unrelated to the patch, so skipping that part now. > [SPS]: Move SPS classes to a separate package > - > > Key: HDFS-12955 > URL: https://issues.apache.org/jira/browse/HDFS-12955 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nn >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Rakesh R >Priority: Trivial > Attachments: HDFS-12955-HDFS-10285-00.patch, > HDFS-12955-HDFS-10285-01.patch > > > For clean modularization, it would be good if we moved SPS related classes > into its own package. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12915) Fix findbugs warning in INodeFile$HeaderFormat.getBlockLayoutRedundancy
[
https://issues.apache.org/jira/browse/HDFS-12915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300908#comment-16300908
]
SammiChen commented on HDFS-12915:
--
bq. IIRC the unit test failed because %02x was printing as a literal, not as a
2-digit hex string using the passed parameter. Replacing an if/throw statement
with a call to a third-party library seems unnecessary. If it's not cleaner in
this case then its appeal, even aesthetically, is limited...
[~chris.douglas], thanks for the further explanation, I'm clear now. Also
thanks for the patch! I didn't realize it before.
bq. On a second thought, I think using ecPolicyID alone is sufficient, so that
we can eliminate blockType as parameter.
[~eddyxu], functionally I agree. While I would suggest to keep the
{{blockType}} for code readability. "0" {{ecPolicyID}} means continuous file
may confuse someone if he/she doesn't know the background.
> Fix findbugs warning in INodeFile$HeaderFormat.getBlockLayoutRedundancy
> ---
>
> Key: HDFS-12915
> URL: https://issues.apache.org/jira/browse/HDFS-12915
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
> Attachments: HDFS-12915.00.patch, HDFS-12915.01.patch
>
>
> It seems HDFS-12840 creates a new findbugs warning.
> Possible null pointer dereference of replication in
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
> Short, Byte)
> Bug type NP_NULL_ON_SOME_PATH (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat
> In method
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
> Short, Byte)
> Value loaded from replication
> Dereferenced at INodeFile.java:[line 210]
> Known null at INodeFile.java:[line 207]
> From a quick look at the patch, it seems bogus though. [~eddyxu][~Sammi]
> would you please double check?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12860) StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300884#comment-16300884
]
genericqa commented on HDFS-12860:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
51s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
24 unchanged - 3 fixed = 27 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 82 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockCountersInPendingIBR |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData |
| | hadoop.hdfs.TestBalancerBandwidth |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12860 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903140/HDFS-12860.00.patch |
| O
[jira] [Updated] (HDFS-12935) Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up
[ https://issues.apache.org/jira/browse/HDFS-12935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianfei Jiang updated HDFS-12935: - Attachment: HDFS-12935.002.patch > Get ambiguous result for DFSAdmin command in HA mode when only one namenode > is up > - > > Key: HDFS-12935 > URL: https://issues.apache.org/jira/browse/HDFS-12935 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 3.0.0-beta1 >Reporter: Jianfei Jiang >Assignee: Jianfei Jiang > Attachments: HDFS-12935.002.patch, HDFS_12935.001.patch > > > In HA mode, if one namenode is down, most of functions can still work. When > considering the following two occasions: > (1)nn1 up and nn2 down > (2)nn1 down and nn2 up > These two occasions should be equivalent. However, some of the DFSAdmin > commands will have ambiguous results. The commands can be send successfully > to the up namenode and are always functionally useful only when nn1 is up > regardless of exception (IOException when connecting to the down namenode > nn2). If only nn2 is up, the commands have no use at all and only exception > to connect nn1 can be found. > See the following command "hdfs dfsadmin setBalancerBandwidth" which aim to > set balancer bandwidth value for datanodes as an example. It works and all > the datanodes can get the setting values only when nn1 is up. If only nn2 is > up, the command throws exception directly and no datanode get the bandwidth > setting. Approximately ten DFSAdmin commands use the similar logical process > and may be ambiguous. > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn1 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 12345 > *Balancer bandwidth is set to 12345 for jiangjianfei01/172.17.0.14:9820* > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei02:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# hdfs haadmin -getServiceState nn2 > active > [root@jiangjianfei01 ~]# hdfs dfsadmin -setBalancerBandwidth 1234 > setBalancerBandwidth: Call From jiangjianfei01/172.17.0.14 to > jiangjianfei01:9820 failed on connection exception: > java.net.ConnectException: Connection refused; For more details see: > http://wiki.apache.org/hadoop/ConnectionRefused > [root@jiangjianfei01 ~]# -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300848#comment-16300848
]
genericqa commented on HDFS-12942:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
45s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 26s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}134m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.tools.TestWebHDFSStoragePolicyCommands |
| | hadoop.hdfs.TestReadWhileWriting |
| | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.TestDFSPermission |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestWriteRead |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.TestDFSStripedOutputStreamWi
[jira] [Commented] (HDFS-12574) Add CryptoInputStream to WebHdfsFileSystem read call.
[
https://issues.apache.org/jira/browse/HDFS-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300811#comment-16300811
]
genericqa commented on HDFS-12574:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
51s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 1s{color} | {color:orange} root: The patch generated 19 new + 392 unchanged
- 2 fixed = 411 total (was 394) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 43s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
37s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 43s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
25s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}136m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}235m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.getRedirectedUrl(Path,
int):in
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$ReadRunner.getRedirectedUrl(Path,
int): String.getBytes() At WebHdfsFileSystem.java:[line 2036] |
| Failed junit tests | hadoop.fs.viewfs.TestViewFileSystemLocalFileSystem |
| | hadoop.fs.viewfs.TestViewFileSystemWithAuthorityLocalFileSystem |
| | hadoop.hdfs.TestHDFSFileSystemContr
[jira] [Commented] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300809#comment-16300809
]
genericqa commented on HDFS-12051:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
16s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 1234 unchanged - 19 fixed = 1236 total (was 1253) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
1s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 1
unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Increment of volatile field
org.apache.hadoop.hdfs.server.namenode.NameCache.size in
org.apache.hadoop.hdfs.server.namenode.NameCache.put(byte[]) At
NameCache.java:in org.apache.hadoop.hdfs.server.namenode.NameCache.put(byte[])
At NameCache.java:[line 116] |
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12051 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903319/HDFS-12051.06.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3f5c528716e5 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision |
[jira] [Commented] (HDFS-12860) StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300788#comment-16300788
]
Lei (Eddy) Xu commented on HDFS-12860:
--
Most test cases are not relevant. Trigger another run of jenkins.
> StripedBlockUtil#getRangesInternalBlocks throws exception for the block group
> size larger than 2GB
> --
>
> Key: HDFS-12860
> URL: https://issues.apache.org/jira/browse/HDFS-12860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12860.00.patch
>
>
> Running terasort on a cluster with 8 datanodes, 256GB data, using
> RS-3-2-1024k.
> The test data was generated by {{teragen}} with 32 mappers.
> The terasort benchmark fails with the following stack trace:
> {code}
> 17/11/27 14:44:31 INFO mapreduce.Job: map 45% reduce 0%
> 17/11/27 14:44:33 INFO mapreduce.Job: Task Id :
> attempt_1510080297865_0160_m_08_0, Status : FAILED
> Error: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil$VerticalRange.(StripedBlockUtil.java:701)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getRangesForInternalBlocks(StripedBlockUtil.java:442)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.divideOneStripe(StripedBlockUtil.java:311)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:308)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:562)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12959) Fix TestOpenFilesWithSnapshot redundant configurations
[
https://issues.apache.org/jira/browse/HDFS-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-12959:
--
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
> Fix TestOpenFilesWithSnapshot redundant configurations
> --
>
> Key: HDFS-12959
> URL: https://issues.apache.org/jira/browse/HDFS-12959
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Minor
> Fix For: 3.1.0
>
> Attachments: HDFS-12959.01.patch
>
>
> Fix the redundant configurations that are set in
> {{TestOpenFilesWithSnapshot#testPointInTimeSnapshotCopiesForOpenFiles}} and
> {{TestOpenFilesWithSnapshot#testOpenFilesSnapChecksumWithTrunkAndAppend}}.
> These redundant configurations give an impression that they are needed for
> the tests to pass through, but infact its not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12959) Fix TestOpenFilesWithSnapshot redundant configurations
[
https://issues.apache.org/jira/browse/HDFS-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300747#comment-16300747
]
Hudson commented on HDFS-12959:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13419 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13419/])
HDFS-12959. Fix TestOpenFilesWithSnapshot redundant configurations. (manojpec:
rev 76e664e931bf0784620b69bc588bd51cf2a024e6)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestOpenFilesWithSnapshot.java
> Fix TestOpenFilesWithSnapshot redundant configurations
> --
>
> Key: HDFS-12959
> URL: https://issues.apache.org/jira/browse/HDFS-12959
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HDFS-12959.01.patch
>
>
> Fix the redundant configurations that are set in
> {{TestOpenFilesWithSnapshot#testPointInTimeSnapshotCopiesForOpenFiles}} and
> {{TestOpenFilesWithSnapshot#testOpenFilesSnapChecksumWithTrunkAndAppend}}.
> These redundant configurations give an impression that they are needed for
> the tests to pass through, but infact its not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12959) Fix TestOpenFilesWithSnapshot redundant configurations
[
https://issues.apache.org/jira/browse/HDFS-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300734#comment-16300734
]
Manoj Govindassamy commented on HDFS-12959:
---
Thanks for the review [~eddyxu]. Test failure is not related to the patch.
Pushed the changes to trunk.
> Fix TestOpenFilesWithSnapshot redundant configurations
> --
>
> Key: HDFS-12959
> URL: https://issues.apache.org/jira/browse/HDFS-12959
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HDFS-12959.01.patch
>
>
> Fix the redundant configurations that are set in
> {{TestOpenFilesWithSnapshot#testPointInTimeSnapshotCopiesForOpenFiles}} and
> {{TestOpenFilesWithSnapshot#testOpenFilesSnapChecksumWithTrunkAndAppend}}.
> These redundant configurations give an impression that they are needed for
> the tests to pass through, but infact its not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12959) Fix TestOpenFilesWithSnapshot redundant configurations
[
https://issues.apache.org/jira/browse/HDFS-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300719#comment-16300719
]
genericqa commented on HDFS-12959:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
11s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}113m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 29s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12959 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903308/HDFS-12959.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9f48863e39f8 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 826507c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22491/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22491/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22491/testReport/ |
| Max. process+thread count | 3037 (vs. ulimit of 5000) |
| modules | C: ha
[jira] [Updated] (HDFS-12954) Ozone: Container : Add key versioning support-3
[
https://issues.apache.org/jira/browse/HDFS-12954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12954:
--
Attachment: HDFS-12954-HDFS-7240.001.patch
> Ozone: Container : Add key versioning support-3
> ---
>
> Key: HDFS-12954
> URL: https://issues.apache.org/jira/browse/HDFS-12954
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-12954-HDFS-7240.001.patch
>
>
> A new version of a key is effectively overwriting some consecutive range of
> bytes in the entire key offset range. For each version, we need to keep
> exactly what the range is in order for the IO vector to work.
> Currently, since we only write from the start (offset = 0), so offset range
> of a version is only up to the key data size field when the version gets
> committed. But currently we only keep one single key data size variable.(see
> {{KeyManagerImpl#commitKey}}). We need to know the corresponding key data
> size for each version. This JIRA is to the tracking of offset range for each
> version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-12954) Ozone: Container : Add key versioning support-3
[
https://issues.apache.org/jira/browse/HDFS-12954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-12954 started by Chen Liang.
-
> Ozone: Container : Add key versioning support-3
> ---
>
> Key: HDFS-12954
> URL: https://issues.apache.org/jira/browse/HDFS-12954
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-12954-HDFS-7240.001.patch
>
>
> A new version of a key is effectively overwriting some consecutive range of
> bytes in the entire key offset range. For each version, we need to keep
> exactly what the range is in order for the IO vector to work.
> Currently, since we only write from the start (offset = 0), so offset range
> of a version is only up to the key data size field when the version gets
> committed. But currently we only keep one single key data size variable.(see
> {{KeyManagerImpl#commitKey}}). We need to know the corresponding key data
> size for each version. This JIRA is to the tracking of offset range for each
> version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300682#comment-16300682 ] Arpit Agarwal commented on HDFS-12942: -- +1 pending Jenkins. > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch, HDFS-12942.004.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300676#comment-16300676 ] Ajay Kumar commented on HDFS-12942: --- [~virajith],[~arpitagarwal] thanks for review. Addressed Arpit's comment in patch v4. > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch, HDFS-12942.004.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12942: -- Attachment: HDFS-12942.004.patch > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch, HDFS-12942.004.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12959) Fix TestOpenFilesWithSnapshot redundant configurations
[
https://issues.apache.org/jira/browse/HDFS-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300669#comment-16300669
]
Lei (Eddy) Xu commented on HDFS-12959:
--
+1. Pending jenkins.
Thanks [~manojg]
> Fix TestOpenFilesWithSnapshot redundant configurations
> --
>
> Key: HDFS-12959
> URL: https://issues.apache.org/jira/browse/HDFS-12959
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HDFS-12959.01.patch
>
>
> Fix the redundant configurations that are set in
> {{TestOpenFilesWithSnapshot#testPointInTimeSnapshotCopiesForOpenFiles}} and
> {{TestOpenFilesWithSnapshot#testOpenFilesSnapChecksumWithTrunkAndAppend}}.
> These redundant configurations give an impression that they are needed for
> the tests to pass through, but infact its not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12958) Ozone: remove setAllocatedBytes method in ContainerInfo
[
https://issues.apache.org/jira/browse/HDFS-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300670#comment-16300670
]
genericqa commented on HDFS-12958:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 1
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.hdfs.TestErasure
[jira] [Updated] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Status: Patch Available (was: In Progress)
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.gro
[jira] [Updated] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Attachment: HDFS-12051.06.patch
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
[jira] [Commented] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300664#comment-16300664
]
Misha Dmitriev commented on HDFS-12051:
---
Addressed Yongjun's comments and submitted a new patch.
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch,
> HDFS-12051.06.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
[jira] [Updated] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HDFS-12051:
--
Status: In Progress (was: Patch Available)
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$LinkedElement[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: o
[jira] [Commented] (HDFS-12943) Consistent Reads from Standby Node
[ https://issues.apache.org/jira/browse/HDFS-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300661#comment-16300661 ] Konstantin Shvachko commented on HDFS-12943: ??what's the function of the Observer Nodes?? Good question. The design doc says that Observer Node is an SBN that does not do checkpoints. Checkpointing degrades performance of SBN, we wont be able to read from it when it's busy. So it's more like a term to distinguish the node which is dedicated for reading - the read-only SBN. Regular SBN is also needed though if we want checkpointing and HA on the cluster, which I do. In the "Note on HA" we talk about some failover scenarios, that reading from ObserverNode elevates it role on the cluster so that you may need to run multiple of them to sustain the response rate in case of failure. > Consistent Reads from Standby Node > -- > > Key: HDFS-12943 > URL: https://issues.apache.org/jira/browse/HDFS-12943 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Reporter: Konstantin Shvachko > Attachments: ConsistentReadsFromStandbyNode.pdf > > > StandbyNode in HDFS is a replica of the active NameNode. The states of the > NameNodes are coordinated via the journal. It is natural to consider > StandbyNode as a read-only replica. As with any replicated distributed system > the problem of stale reads should be resolved. Our main goal is to provide > reads from standby in a consistent way in order to enable a wide range of > existing applications running on top of HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[
https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300648#comment-16300648
]
Arpit Agarwal commented on HDFS-12942:
--
The v3 patch looks good to me. Very minor comment on the unit test
{{testMoveBlockFailure}}. The following log message should say
testMoveBlockFailure.
{code}
} catch (Exception ex) {
LOG.info("Exception in testMoveBlockSuccess ", ex);
{code}
+1 otherwise.
> Synchronization issue in FSDataSetImpl#moveBlock
>
>
> Key: HDFS-12942
> URL: https://issues.apache.org/jira/browse/HDFS-12942
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch,
> HDFS-12942.003.patch
>
>
> FSDataSetImpl#moveBlock works in following following 3 steps:
> # first creates a new replicaInfo object
> # calls finalizeReplica to finalize it.
> # Calls removeOldReplica to remove oldReplica.
> A client can potentially append to the old replica between step 1 and 2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11847) Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300643#comment-16300643
]
genericqa commented on HDFS-11847:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
42s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 51s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
810 unchanged - 1 fixed = 811 total (was 811) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}120m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-11847 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903299/HDFS-11847.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall m
[jira] [Commented] (HDFS-12165) getSnapshotDiffReport throws NegativeArraySizeException for very large snapshot diff summary
[
https://issues.apache.org/jira/browse/HDFS-12165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300603#comment-16300603
]
Wei-Chiu Chuang commented on HDFS-12165:
It seems to me HDFS-12594 is a possible for fix this issue.
> getSnapshotDiffReport throws NegativeArraySizeException for very large
> snapshot diff summary
>
>
> Key: HDFS-12165
> URL: https://issues.apache.org/jira/browse/HDFS-12165
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wei-Chiu Chuang
>
> For a really large snapshot diff, getSnapshotDiffReport throws
> NegativeArraySizeException
> {noformat}
> 2017-07-19 11:14:16,415 WARN org.apache.hadoop.ipc.Server: Error serializing
> call response for call
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getSnapshotDiffReport
> from 10.17.211.10:58223 Call#0 Retry#0
> java.lang.NegativeArraySizeException
> at
> com.google.protobuf.CodedOutputStream.newInstance(CodedOutputStream.java:105)
> at
> com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:87)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$RpcResponseWrapper.write(ProtobufRpcEngine.java:468)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2410)
> at org.apache.hadoop.ipc.Server.access$500(Server.java:134)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2182)
> {noformat}
> This particular snapshot diff contains more than 25 million different file
> system objects, and which means the serialized response can be more than 2GB,
> overflowing protobuf length calculation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12957) Multi master name node
[ https://issues.apache.org/jira/browse/HDFS-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300592#comment-16300592 ] Andrew Schiefelbein edited comment on HDFS-12957 at 12/21/17 10:02 PM: --- Thank you for the comments. I do understand that the ask is not a trivial one and I'm doing my best to try to optimize the current setup I have for throughput. The choices for my particular use case is to buy an expensive san / nas, use a glusterfs style setup or to hope that I can get enough throughput on the name node. The scale of reads I need to do I'm not sure I can accomplish the speed I need in a system that has to go through a single node, what I'm trying to do may be an edge case and not fit within the HDFS model. While I can get the HA setup with active / passive to work and I can have it behind a load balancer with a health check, because the fail over should be transparent to the down stream processes, I would much prefer to have an active / active setup instead. Even if it's 2 name nodes, both of them active at the same time to me is a better solution than having one in a passive role. If we're going to spend the time and expense to setup servers to do a role they should take load or it's a waste of electricity to sit in a passive mode for the "what if" days that may never come. It's an opinion and I know what that means, but if it could be done I think it would be a wonderful addition to the tool set. was (Author: aschiefe): Thank you for the comments. I do understand that the ask is not a trivial one and I'm doing my best to try to optimize the current setup I have for throughput. The choices for my particular use case is to buy an expensive san / nas, use a glusterfs style setup or to hope that I can get enough throughput on the name node. The scale of reads I need to do I'm not sure I can accomplish the speed I need in a system that has to go through a single node, what I'm trying to do may be an edge case and not fit within the HDFS model. While I can get the HA setup with active / passive to work and I can have it behind a load balancer with a health check, because the fail over should be transparent to the down stream processes, I would much prefer not to have an active / active setup instead. Even if it's 2 name nodes, both of them active at the same time to me is a better solution than having one in a passive role. If we're going to spend the time and expense to setup servers to do a role they should take load or it's a waste of electricity to sit in a passive mode for the "what if" days that may never come. It's an opinion and I know what that means, but if it could be done I think it would be a wonderful addition to the tool set. > Multi master name node > -- > > Key: HDFS-12957 > URL: https://issues.apache.org/jira/browse/HDFS-12957 > Project: Hadoop HDFS > Issue Type: Wish > Components: namenode >Affects Versions: 3.0.0 > Environment: Multi node high availability with high throughput >Reporter: Andrew Schiefelbein >Priority: Minor > > While I do appreciate that the HA and federated setups do take care of some > of the issues running in a multi node distributed fault tolerant way I > believe that a singe name node is a bit of a pain point for me. > My wish, and your mileage may vary, is that here could be a name node on each > data node system so that each system could act as a gateway and worker for > the cluster. > In my tests right now I can push nearly 2 million records / second through a > single node instance, but when I bring up 4 more nodes it doesn't go 4x > faster as it has to retrieve from the name node and go through it as a > gateway to the cluster. For speed and resiliency if this could be spread out > among n number of nodes and put behind a load balancer it would be the ideal > solution for distributed resilient and high throughput shared storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12957) Multi master name node
[ https://issues.apache.org/jira/browse/HDFS-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300592#comment-16300592 ] Andrew Schiefelbein edited comment on HDFS-12957 at 12/21/17 10:01 PM: --- Thank you for the comments. I do understand that the ask is not a trivial one and I'm doing my best to try to optimize the current setup I have for throughput. The choices for my particular use case is to buy an expensive san / nas, use a glusterfs style setup or to hope that I can get enough throughput on the name node. The scale of reads I need to do I'm not sure I can accomplish the speed I need in a system that has to go through a single node, what I'm trying to do may be an edge case and not fit within the HDFS model. While I can get the HA setup with active / passive to work and I can have it behind a load balancer with a health check, because the fail over should be transparent to the down stream processes, I would much prefer not to have an active / active setup instead. Even if it's 2 name nodes, both of them active at the same time to me is a better solution than having one in a passive role. If we're going to spend the time and expense to setup servers to do a role they should take load or it's a waste of electricity to sit in a passive mode for the "what if" days that may never come. It's an opinion and I know what that means, but if it could be done I think it would be a wonderful addition to the tool set. was (Author: aschiefe): Thank you for the comments. I do understand that the ask is not a trivial one and I'm doing my best to try to optimize the current setup I have for throughput. The choices for my particular use case is to buy an expensive san / nas, use a glusterfs style setup or to hope that I can get enough throughput on the name node. The scale of reads I need to do I'm not sure I can accomplish the speed I need in a system that has to go through a single node, what I'm trying to do may be an edge case and not fit within the HDFS model. While I can get the HA setup with active / passive to work and I can have it behind a load balancer with a health check because the fail over should be transparent to the down stream processes I would much prefer not to have an active / active setup instead. Even if it's 2 name nodes, both of them active at the same time to me is a better solution than having one in a passive role. If we're going to spend the time and expense to setup servers to do a role they should take load or it's a waste of electricity to sit in a passive mode for the "what if" days that may never come. It's an opinion and I know what that means, but if it could be done I think it would be a wonderful addition to the tool set. > Multi master name node > -- > > Key: HDFS-12957 > URL: https://issues.apache.org/jira/browse/HDFS-12957 > Project: Hadoop HDFS > Issue Type: Wish > Components: namenode >Affects Versions: 3.0.0 > Environment: Multi node high availability with high throughput >Reporter: Andrew Schiefelbein >Priority: Minor > > While I do appreciate that the HA and federated setups do take care of some > of the issues running in a multi node distributed fault tolerant way I > believe that a singe name node is a bit of a pain point for me. > My wish, and your mileage may vary, is that here could be a name node on each > data node system so that each system could act as a gateway and worker for > the cluster. > In my tests right now I can push nearly 2 million records / second through a > single node instance, but when I bring up 4 more nodes it doesn't go 4x > faster as it has to retrieve from the name node and go through it as a > gateway to the cluster. For speed and resiliency if this could be spread out > among n number of nodes and put behind a load balancer it would be the ideal > solution for distributed resilient and high throughput shared storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12957) Multi master name node
[ https://issues.apache.org/jira/browse/HDFS-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300592#comment-16300592 ] Andrew Schiefelbein commented on HDFS-12957: Thank you for the comments. I do understand that the ask is not a trivial one and I'm doing my best to try to optimize the current setup I have for throughput. The choices for my particular use case is to buy an expensive san / nas, use a glusterfs style setup or to hope that I can get enough throughput on the name node. The scale of reads I need to do I'm not sure I can accomplish the speed I need in a system that has to go through a single node, what I'm trying to do may be an edge case and not fit within the HDFS model. While I can get the HA setup with active / passive to work and I can have it behind a load balancer with a health check because the fail over should be transparent to the down stream processes I would much prefer not to have an active / active setup instead. Even if it's 2 name nodes, both of them active at the same time to me is a better solution than having one in a passive role. If we're going to spend the time and expense to setup servers to do a role they should take load or it's a waste of electricity to sit in a passive mode for the "what if" days that may never come. It's an opinion and I know what that means, but if it could be done I think it would be a wonderful addition to the tool set. > Multi master name node > -- > > Key: HDFS-12957 > URL: https://issues.apache.org/jira/browse/HDFS-12957 > Project: Hadoop HDFS > Issue Type: Wish > Components: namenode >Affects Versions: 3.0.0 > Environment: Multi node high availability with high throughput >Reporter: Andrew Schiefelbein >Priority: Minor > > While I do appreciate that the HA and federated setups do take care of some > of the issues running in a multi node distributed fault tolerant way I > believe that a singe name node is a bit of a pain point for me. > My wish, and your mileage may vary, is that here could be a name node on each > data node system so that each system could act as a gateway and worker for > the cluster. > In my tests right now I can push nearly 2 million records / second through a > single node instance, but when I bring up 4 more nodes it doesn't go 4x > faster as it has to retrieve from the name node and go through it as a > gateway to the cluster. For speed and resiliency if this could be spread out > among n number of nodes and put behind a load balancer it would be the ideal > solution for distributed resilient and high throughput shared storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300589#comment-16300589 ] Virajith Jalaparti commented on HDFS-12942: --- +1 on patch v3. > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[
https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300587#comment-16300587
]
genericqa commented on HDFS-12948:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
5s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 33s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}120m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903296/HDFS-12948.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 44a4e50ef4bf 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b318bed |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/22487/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/22487/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22487/testReport/ |
| Max. process+thread co
[jira] [Commented] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[
https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300564#comment-16300564
]
genericqa commented on HDFS-12948:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
1s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 58s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
24s{color} | {color:red} The patch generated 10 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart |
| | hadoop.hdfs.server.namenode.TestAuditLoggerWithCommands |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.TestProtectedDirectories |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport |
| | hadoop.hdfs.server.namenode.TestNestedEncryptionZones |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12948 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903300/HDFS-12948.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 91712e7377f2 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b318bed |
| maven
[jira] [Updated] (HDFS-12574) Add CryptoInputStream to WebHdfsFileSystem read call.
[ https://issues.apache.org/jira/browse/HDFS-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-12574: -- Attachment: HDFS-12574.007.patch > Add CryptoInputStream to WebHdfsFileSystem read call. > - > > Key: HDFS-12574 > URL: https://issues.apache.org/jira/browse/HDFS-12574 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption, kms, webhdfs >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah > Attachments: HDFS-12574.001.patch, HDFS-12574.002.patch, > HDFS-12574.003.patch, HDFS-12574.004.patch, HDFS-12574.005.patch, > HDFS-12574.006.patch, HDFS-12574.007.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12942) Synchronization issue in FSDataSetImpl#moveBlock
[ https://issues.apache.org/jira/browse/HDFS-12942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300549#comment-16300549 ] Ajay Kumar commented on HDFS-12942: --- findbug warning and failed tests are unrelated.(all of them passed locally) > Synchronization issue in FSDataSetImpl#moveBlock > > > Key: HDFS-12942 > URL: https://issues.apache.org/jira/browse/HDFS-12942 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12942.001.patch, HDFS-12942.002.patch, > HDFS-12942.003.patch > > > FSDataSetImpl#moveBlock works in following following 3 steps: > # first creates a new replicaInfo object > # calls finalizeReplica to finalize it. > # Calls removeOldReplica to remove oldReplica. > A client can potentially append to the old replica between step 1 and 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12888) NameNode web UI shows stale config values after cli refresh
[
https://issues.apache.org/jira/browse/HDFS-12888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar reassigned HDFS-12888:
-
Assignee: Ajay Kumar
> NameNode web UI shows stale config values after cli refresh
> ---
>
> Key: HDFS-12888
> URL: https://issues.apache.org/jira/browse/HDFS-12888
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ui
>Affects Versions: 2.7.4
>Reporter: Zhe Zhang
>Assignee: Ajay Kumar
>
> To reproduce:
> # Run webui /conf
> # Use {{hdfs --refreshSuperUserGroupsConfiguration}} to update a
> configuration value
> # Run webui /conf again, it will still show the old configuration value
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12959) Fix TestOpenFilesWithSnapshot redundant configurations
[
https://issues.apache.org/jira/browse/HDFS-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-12959:
--
Status: Patch Available (was: Open)
> Fix TestOpenFilesWithSnapshot redundant configurations
> --
>
> Key: HDFS-12959
> URL: https://issues.apache.org/jira/browse/HDFS-12959
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HDFS-12959.01.patch
>
>
> Fix the redundant configurations that are set in
> {{TestOpenFilesWithSnapshot#testPointInTimeSnapshotCopiesForOpenFiles}} and
> {{TestOpenFilesWithSnapshot#testOpenFilesSnapChecksumWithTrunkAndAppend}}.
> These redundant configurations give an impression that they are needed for
> the tests to pass through, but infact its not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12959) Fix TestOpenFilesWithSnapshot redundant configurations
[
https://issues.apache.org/jira/browse/HDFS-12959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-12959:
--
Attachment: HDFS-12959.01.patch
Attached v01 patch to remove the redundant configurations in
TestOpenFilesWithSnapshot.
[~eddyxu], can you please take a look at the patch?
> Fix TestOpenFilesWithSnapshot redundant configurations
> --
>
> Key: HDFS-12959
> URL: https://issues.apache.org/jira/browse/HDFS-12959
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Minor
> Attachments: HDFS-12959.01.patch
>
>
> Fix the redundant configurations that are set in
> {{TestOpenFilesWithSnapshot#testPointInTimeSnapshotCopiesForOpenFiles}} and
> {{TestOpenFilesWithSnapshot#testOpenFilesSnapChecksumWithTrunkAndAppend}}.
> These redundant configurations give an impression that they are needed for
> the tests to pass through, but infact its not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12051) Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
[
https://issues.apache.org/jira/browse/HDFS-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300531#comment-16300531
]
Yongjun Zhang commented on HDFS-12051:
--
Thanks for the updated version Misha.
Couple of minor comments,
1. the const 5 is better defined as a constant in the class,
{{for (int collisnChainLen = 0; collisnChainLen < 5; collisnChainLen++) {}}
2. The value in {{private final static int DEFAULT_CACHE_SIZE = 4 * 1024 *
1024;}} should be referencing DFS_NAMENODE_NAME_CACHE_SIZE_DEFAULT instead of
hardcode again here.
Hi [~daryn] and [~kihwal],
The fix [~mi...@cloudera.com] did here is a good improvement. Probably slight
concern is about run time computation cost. Prior to this change, all INode
names go through this computation. Misha added some more for improving memory
usage better.
The patch looks good to me over all. Wonder if you guys have further thoughts?
Thanks.
> Intern INOdeFileAttributes$SnapshotCopy.name byte[] arrays to save memory
> -
>
> Key: HDFS-12051
> URL: https://issues.apache.org/jira/browse/HDFS-12051
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
> Attachments: HDFS-12051.01.patch, HDFS-12051.02.patch,
> HDFS-12051.03.patch, HDFS-12051.04.patch, HDFS-12051.05.patch
>
>
> When snapshot diff operation is performed in a NameNode that manages several
> million HDFS files/directories, NN needs a lot of memory. Analyzing one heap
> dump with jxray (www.jxray.com), we observed that duplicate byte[] arrays
> result in 6.5% memory overhead, and most of these arrays are referenced by
> {{org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name}}
> and {{org.apache.hadoop.hdfs.server.namenode.INodeFile.name}}:
> {code}
> 19. DUPLICATE PRIMITIVE ARRAYS
> Types of duplicate objects:
> Ovhd Num objs Num unique objs Class name
> 3,220,272K (6.5%) 104749528 25760871 byte[]
>
> 1,841,485K (3.7%), 53194037 dup arrays (13158094 unique)
> 3510556 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 2228255
> of byte[8](48, 48, 48, 48, 48, 48, 95, 48), 357439 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 237395 of byte[8](48, 48, 48, 48, 48, 49,
> 95, 48), 227853 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...),
> 179193 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...), 169487
> of byte[8](48, 48, 48, 48, 48, 50, 95, 48), 145055 of byte[17](112, 97, 114,
> 116, 45, 109, 45, 48, 48, 48, ...), 128134 of byte[8](48, 48, 48, 48, 48, 51,
> 95, 48), 108265 of byte[17](112, 97, 114, 116, 45, 109, 45, 48, 48, 48, ...)
> ... and 45902395 more arrays, of which 13158084 are unique
> <--
> org.apache.hadoop.hdfs.server.namenode.INodeFileAttributes$SnapshotCopy.name
> <-- org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiff.snapshotINode
> <-- {j.u.ArrayList} <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileDiffList.diffs <--
> org.apache.hadoop.hdfs.server.namenode.snapshot.FileWithSnapshotFeature.diffs
> <-- org.apache.hadoop.hdfs.server.namenode.INode$Feature[] <--
> org.apache.hadoop.hdfs.server.namenode.INodeFile.features <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <-- ... (1
> elements) ... <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap$1.entries <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.blocks <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.blocksMap <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockReportProcessingThread.this$0
> <-- j.l.Thread[] <-- j.l.ThreadGroup.threads <-- j.l.Thread.group <-- Java
> Static: org.apache.hadoop.fs.FileSystem$Statistics.STATS_DATA_CLEANER
> 409,830K (0.8%), 13482787 dup arrays (13260241 unique)
> 430 of byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 353 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 352 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 350 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 342 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 341 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 340 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 337 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...), 334 of
> byte[32](116, 97, 115, 107, 95, 49, 52, 57, 55, 48, ...)
> ... and 13479257 more arrays, of which 13260231 are unique
> <-- org.apache.hadoop.hdfs.server.namenode.INodeFile.name <--
> org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.bc <--
> org.apache.hadoop.util.LightWeightGSet$Linke
[jira] [Commented] (HDFS-12947) Limit the number of Snapshots allowed to be created for a Snapshottable Directory
[
https://issues.apache.org/jira/browse/HDFS-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300529#comment-16300529
]
genericqa commented on HDFS-12947:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
52s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 441 unchanged - 1 fixed = 441 total (was 442) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 24s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.server.namenode.snapshot.TestRandomOpsWithSnapshots |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.namenode.snapshot.TestAclWithSnapshot |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12947 |
| JIRA Patch URL |
https://issues.apache.org/j
[jira] [Created] (HDFS-12959) Fix TestOpenFilesWithSnapshot redundant configurations
Manoj Govindassamy created HDFS-12959:
-
Summary: Fix TestOpenFilesWithSnapshot redundant configurations
Key: HDFS-12959
URL: https://issues.apache.org/jira/browse/HDFS-12959
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.0.0
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
Priority: Minor
Fix the redundant configurations that are set in
{{TestOpenFilesWithSnapshot#testPointInTimeSnapshotCopiesForOpenFiles}} and
{{TestOpenFilesWithSnapshot#testOpenFilesSnapChecksumWithTrunkAndAppend}}.
These redundant configurations give an impression that they are needed for the
tests to pass through, but infact its not.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300516#comment-16300516 ] Virajith Jalaparti edited comment on HDFS-10285 at 12/21/17 8:16 PM: - Hi [~umamaheswararao], Thanks for the meeting summary. Having both the options (SPS within NN and SPS as service) would be great. bq. it may be necessary to start SPS RPC with its own IP:port (within NN or outside), so clients can always talk to SPS on that port, irrespective of where its running. Quick question about this -- what are clients you are referring to here? Are these for admin operations? If the SPS is going to run outside the NN, I would think it is going to be decoupled from/not depend on the FSNamesystem lock and the NN-DN heartbeat protocol. The current implementation/design has a tight coupling between the SPS and both these components. was (Author: virajith): Hi [~umamaheswararao], Thanks for the meeting summary. Having both the options (SPS within NN and SPS as service) would be great. bq. it may be necessary to start SPS RPC with its own IP:port (within NN or outside), so clients can always talk to SPS on that port, irrespective of where its running. Quick question about this: what are clients you are referring to here? Are these for admin operations? If the SPS is going to run outside the NN, I would think it is going to be decoupled from/not depend on the FSNamesystem lock and the NN-DN heartbeat protocol. The current implementation/design has a tight coupling between the SPS and both these components. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300516#comment-16300516 ] Virajith Jalaparti edited comment on HDFS-10285 at 12/21/17 8:15 PM: - Hi [~umamaheswararao], Thanks for the meeting summary. Having both the options (SPS within NN and SPS as service) would be great. bq. it may be necessary to start SPS RPC with its own IP:port (within NN or outside), so clients can always talk to SPS on that port, irrespective of where its running. Quick question about this: what are clients you are referring to here? Are these for admin operations? If the SPS is going to run outside the NN, I would think it is going to be decoupled from/not depend on the FSNamesystem lock and the NN-DN heartbeat protocol. The current implementation/design has a tight coupling between the SPS and both these components. was (Author: virajith): Hi [~umamaheswararao], Thanks for the meeting summary. I agree that having both the options (SPS within NN and SPS as service) would be great to have. bq. it may be necessary to start SPS RPC with its own IP:port (within NN or outside), so clients can always talk to SPS on that port, irrespective of where its running. Quick question about this: what are clients you are referring to here? Are these for admin operations? If the SPS is going to run outside the NN, I would think it is going to be decoupled from/not depend on the FSNamesystem lock and the NN-DN heartbeat protocol. The current implementation/design has a tight coupling between the SPS and both these components. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300516#comment-16300516 ] Virajith Jalaparti commented on HDFS-10285: --- Hi [~umamaheswararao], Thanks for the meeting summary. I agree that having both the options (SPS within NN and SPS as service) would be great to have. bq. it may be necessary to start SPS RPC with its own IP:port (within NN or outside), so clients can always talk to SPS on that port, irrespective of where its running. Quick question about this: what are clients you are referring to here? Are these for admin operations? If the SPS is going to run outside the NN, I would think it is going to be decoupled from/not depend on the FSNamesystem lock and the NN-DN heartbeat protocol. The current implementation/design has a tight coupling between the SPS and both these components. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12951) Incorrect javadoc in SaslDataTransferServer.java#receive
[
https://issues.apache.org/jira/browse/HDFS-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300478#comment-16300478
]
Hudson commented on HDFS-12951:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13418 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13418/])
HDFS-12951. Incorrect javadoc in SaslDataTransferServer.java#receive. (cliang:
rev 826507c41b7dd89ce5b53d2245d09c2443423670)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
> Incorrect javadoc in SaslDataTransferServer.java#receive
> -
>
> Key: HDFS-12951
> URL: https://issues.apache.org/jira/browse/HDFS-12951
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HDFS-12951.001.patch
>
>
> The javadoc for the receive incorrectly states "int" in the 4th parameter.
> This should be corrected to remove the int.
> {code}
> /**
>* Receives SASL negotiation from a peer on behalf of a server.
>*
>* @param peer connection peer
>* @param underlyingOut connection output stream
>* @param underlyingIn connection input stream
>* @param int xferPort data transfer port of DataNode accepting connection
>* @param datanodeId ID of DataNode accepting connection
>* @return new pair of streams, wrapped after SASL negotiation
>* @throws IOException for any error
>*/
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12958) Ozone: remove setAllocatedBytes method in ContainerInfo
[
https://issues.apache.org/jira/browse/HDFS-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12958:
--
Attachment: HDFS-12958-HDFS-7240.001.patch
Post v001 patch. [~nandakumar131] would you mind taking a look? Thanks!
> Ozone: remove setAllocatedBytes method in ContainerInfo
> ---
>
> Key: HDFS-12958
> URL: https://issues.apache.org/jira/browse/HDFS-12958
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HDFS-12958-HDFS-7240.001.patch
>
>
> We may want to remove {{setAllocatedBytes}} method from {{ContainerInfo}} and
> we keep all fields of {{ContainerInfo}} immutable, such that client won't
> accidentally change {{ContainerInfo}} and rely on the changed instance.
> An alternative of having {{setAllocatedBytes}} is to always create a new
> {{ContainerInfo}} instance whenever it needs to be changed.
> This is based on [this
> comment|https://issues.apache.org/jira/browse/HDFS-12751?focusedCommentId=16299750&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16299750]
> from HDFS-12751.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12958) Ozone: remove setAllocatedBytes method in ContainerInfo
[
https://issues.apache.org/jira/browse/HDFS-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12958:
--
Status: Patch Available (was: Open)
> Ozone: remove setAllocatedBytes method in ContainerInfo
> ---
>
> Key: HDFS-12958
> URL: https://issues.apache.org/jira/browse/HDFS-12958
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Minor
> Attachments: HDFS-12958-HDFS-7240.001.patch
>
>
> We may want to remove {{setAllocatedBytes}} method from {{ContainerInfo}} and
> we keep all fields of {{ContainerInfo}} immutable, such that client won't
> accidentally change {{ContainerInfo}} and rely on the changed instance.
> An alternative of having {{setAllocatedBytes}} is to always create a new
> {{ContainerInfo}} instance whenever it needs to be changed.
> This is based on [this
> comment|https://issues.apache.org/jira/browse/HDFS-12751?focusedCommentId=16299750&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16299750]
> from HDFS-12751.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12951) Incorrect javadoc in SaslDataTransferServer.java#receive
[
https://issues.apache.org/jira/browse/HDFS-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-12951:
--
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
> Incorrect javadoc in SaslDataTransferServer.java#receive
> -
>
> Key: HDFS-12951
> URL: https://issues.apache.org/jira/browse/HDFS-12951
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HDFS-12951.001.patch
>
>
> The javadoc for the receive incorrectly states "int" in the 4th parameter.
> This should be corrected to remove the int.
> {code}
> /**
>* Receives SASL negotiation from a peer on behalf of a server.
>*
>* @param peer connection peer
>* @param underlyingOut connection output stream
>* @param underlyingIn connection input stream
>* @param int xferPort data transfer port of DataNode accepting connection
>* @param datanodeId ID of DataNode accepting connection
>* @return new pair of streams, wrapped after SASL negotiation
>* @throws IOException for any error
>*/
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12951) Incorrect javadoc in SaslDataTransferServer.java#receive
[
https://issues.apache.org/jira/browse/HDFS-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300466#comment-16300466
]
Chen Liang commented on HDFS-12951:
---
Thanks [~msingh] for the catch, the failed tests and findbug warnings are
unrelated. +1 on v001 patch, I've committed to trunk. Thanks for the
contribution.
> Incorrect javadoc in SaslDataTransferServer.java#receive
> -
>
> Key: HDFS-12951
> URL: https://issues.apache.org/jira/browse/HDFS-12951
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: 3.1.0
>
> Attachments: HDFS-12951.001.patch
>
>
> The javadoc for the receive incorrectly states "int" in the 4th parameter.
> This should be corrected to remove the int.
> {code}
> /**
>* Receives SASL negotiation from a peer on behalf of a server.
>*
>* @param peer connection peer
>* @param underlyingOut connection output stream
>* @param underlyingIn connection input stream
>* @param int xferPort data transfer port of DataNode accepting connection
>* @param datanodeId ID of DataNode accepting connection
>* @return new pair of streams, wrapped after SASL negotiation
>* @throws IOException for any error
>*/
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12948: --- Status: Patch Available (was: Open) > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Attachments: HDFS-12948.001.patch > > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12948: --- Attachment: HDFS-12948.001.patch > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Attachments: HDFS-12948.001.patch > > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12948: --- Status: Open (was: Patch Available) > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12948: --- Attachment: (was: HDFS-12948.001.patch) > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12915) Fix findbugs warning in INodeFile$HeaderFormat.getBlockLayoutRedundancy
[
https://issues.apache.org/jira/browse/HDFS-12915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300446#comment-16300446
]
Lei (Eddy) Xu commented on HDFS-12915:
--
Thanks for the patch, [~chris.douglas]. It LGTM overall.
bq. If erasureCodingPolicyID is null and blockType is stripped, it should throw
exception. ...
bq. I also thought that was odd, but it's the existing behavior if it can be
returned by the policy manager.
I think the reason of this is that HDFS supports different EC policies (RS-3-2,
RS-6-3, and etc), and the EC policy is bind to an INodeFile. Giving {{blockType
== striped}} along is not enough to set the EC policy correctly.
On a second thought, I think using {{ecPolicyID}} alone is sufficient, so that
we can eliminate {{blockType}} as parameter. [~Sammi] What do you think?
> Fix findbugs warning in INodeFile$HeaderFormat.getBlockLayoutRedundancy
> ---
>
> Key: HDFS-12915
> URL: https://issues.apache.org/jira/browse/HDFS-12915
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
> Attachments: HDFS-12915.00.patch, HDFS-12915.01.patch
>
>
> It seems HDFS-12840 creates a new findbugs warning.
> Possible null pointer dereference of replication in
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
> Short, Byte)
> Bug type NP_NULL_ON_SOME_PATH (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat
> In method
> org.apache.hadoop.hdfs.server.namenode.INodeFile$HeaderFormat.getBlockLayoutRedundancy(BlockType,
> Short, Byte)
> Value loaded from replication
> Dereferenced at INodeFile.java:[line 210]
> Known null at INodeFile.java:[line 207]
> From a quick look at the patch, it seems bogus though. [~eddyxu][~Sammi]
> would you please double check?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12958) Ozone: remove setAllocatedBytes method in ContainerInfo
[
https://issues.apache.org/jira/browse/HDFS-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang reassigned HDFS-12958:
-
Assignee: Chen Liang
> Ozone: remove setAllocatedBytes method in ContainerInfo
> ---
>
> Key: HDFS-12958
> URL: https://issues.apache.org/jira/browse/HDFS-12958
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Minor
>
> We may want to remove {{setAllocatedBytes}} method from {{ContainerInfo}} and
> we keep all fields of {{ContainerInfo}} immutable, such that client won't
> accidentally change {{ContainerInfo}} and rely on the changed instance.
> An alternative of having {{setAllocatedBytes}} is to always create a new
> {{ContainerInfo}} instance whenever it needs to be changed.
> This is based on [this
> comment|https://issues.apache.org/jira/browse/HDFS-12751?focusedCommentId=16299750&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16299750]
> from HDFS-12751.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11847) Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning
[
https://issues.apache.org/jira/browse/HDFS-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11847:
--
Attachment: HDFS-11847.04.patch
Attached v04 patch to address the TestAnnotations failure in the previous test
run. Other test failures are not related to the patch.
> Enhance dfsadmin listOpenFiles command to list files blocking datanode
> decommissioning
> --
>
> Key: HDFS-11847
> URL: https://issues.apache.org/jira/browse/HDFS-11847
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HDFS-11847.01.patch, HDFS-11847.02.patch,
> HDFS-11847.03.patch, HDFS-11847.04.patch
>
>
> HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list
> all the open files in the system.
> Additionally, it would be very useful to only list open files that are
> blocking the DataNode decommissioning. With thousand+ node clusters, where
> there might be machines added and removed regularly for maintenance, any
> option to monitor and debug decommissioning status is very helpful. Proposal
> here is to add suboptions to {{listOpenFiles}} for the above case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12957) Multi master name node
[ https://issues.apache.org/jira/browse/HDFS-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300432#comment-16300432 ] Eric Yang commented on HDFS-12957: -- [~aschiefe] The assumption of distributed metadata can provide faster response may not be true in real life. I had a rare opportunity to work on an [implementation of distributed file system|https://en.wikipedia.org/wiki/IBM_General_Parallel_File_System] without namenode, while provide HDFS API compatibility. The metadata is distributed among metadata nodes. File system queries are broadcasted to all metadata nodes to find the answers and the final result is assembled on the file system client side or namenode proxy. All servers need to look up it's internal data structure and transmit data cross network. The result is the latency is almost the same as namespace lookup from single namenode, and most of the time performance is worse than single namenode because it takes longer on slower nodes. In depth analysis shown that, when metadata changes, there are several locking that needs to be coordinated among metadata servers. This is one of the most expensive operation on distributed meatadata system. It is 100 times cheaper to implement locking on the same multi-processors system than attempt to implement locking over network. If you are interested in scaling namenode into distributed namenode, you might want to consider to optimize the workload to use larger file size and larger blocks, and expect each IO operation to take 100x longer to perform than single name node system. The metadata system must be kept in small number of nodes to improve response. Hope this information helps for people who wants to try. > Multi master name node > -- > > Key: HDFS-12957 > URL: https://issues.apache.org/jira/browse/HDFS-12957 > Project: Hadoop HDFS > Issue Type: Wish > Components: namenode >Affects Versions: 3.0.0 > Environment: Multi node high availability with high throughput >Reporter: Andrew Schiefelbein >Priority: Minor > > While I do appreciate that the HA and federated setups do take care of some > of the issues running in a multi node distributed fault tolerant way I > believe that a singe name node is a bit of a pain point for me. > My wish, and your mileage may vary, is that here could be a name node on each > data node system so that each system could act as a gateway and worker for > the cluster. > In my tests right now I can push nearly 2 million records / second through a > single node instance, but when I bring up 4 more nodes it doesn't go 4x > faster as it has to retrieve from the name node and go through it as a > gateway to the cluster. For speed and resiliency if this could be spread out > among n number of nodes and put behind a load balancer it would be the ideal > solution for distributed resilient and high throughput shared storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12938) TestErasureCodigCLI testAll failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300430#comment-16300430
]
Hudson commented on HDFS-12938:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13417 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13417/])
HDFS-12938. TestErasureCodigCLI testAll failing consistently. (lei: rev
b318bed01affa150d70661f263efff9a5c9422f6)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
> TestErasureCodigCLI testAll failing consistently.
> -
>
> Key: HDFS-12938
> URL: https://issues.apache.org/jira/browse/HDFS-12938
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-12938.001.patch
>
>
> {{TestErasureCodingCLI#testAll}} is failing consistently.
> It failed in this precommit:
> https://builds.apache.org/job/PreCommit-HDFS-Build/22435/testReport/org.apache.hadoop.cli/TestErasureCodingCLI/testAll/
> I ran locally on my laptop and it failed too.
> It failed with this stack trace:
> {noformat}
> java.lang.AssertionError: One of the tests failed. See the Detailed results
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:264)
> at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:126)
> at
> org.apache.hadoop.cli.TestErasureCodingCLI.tearDown(TestErasureCodingCLI.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Below is the detailed report from
> {{org.apache.hadoop.cli.TestErasureCodingCLI-output.txt}}.
> {noformat}
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(156)) -
> ---
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(157)) - Test ID: [15]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(158)) -Test Description:
> [setPolicy : set policy on non-empty directory]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(159)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -mkdir /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -touchz /ecdir/file1]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -setPolicy -policy RS-6-3-1024k -path /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(167)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(170)) -Cleanup Commands: [-fs
> hdfs://localhost:52345 -rm -R /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(174)) -
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(178)) - Comparator:
> [SubstringComparator]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(180)) - Comparision result:
> [fail]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(182)) - Expected output:
> [Warning: setting erasure coding policy on an non-empty directory will not
> automatically convert existing data to RS-6-3-1024]
> 2017-12-18 09:25:44,822 [Threa
[jira] [Updated] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12948: --- Attachment: HDFS-12948.001.patch Patch v1 handles the case where if the top limit specified for report command is not valid, it throws an exception. [~anu], please have a look. > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Attachments: HDFS-12948.001.patch > > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12948) DiskBalancer report command top option should only take positive numeric values
[ https://issues.apache.org/jira/browse/HDFS-12948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12948: --- Status: Patch Available (was: Open) > DiskBalancer report command top option should only take positive numeric > values > --- > > Key: HDFS-12948 > URL: https://issues.apache.org/jira/browse/HDFS-12948 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Minor > Attachments: HDFS-12948.001.patch > > > Currently , diskBalancer report command "top" option takes values like "AAA" > as well negative/decimal values and gives output. These invalid values should > not be processed and some error/warning should be given. > For Example, > $ hdfs diskbalancer -report -top -100 > 17/12/19 15:07:01 INFO command.Command: Processing report command > 17/12/19 15:07:02 INFO balancer.KeyManager: Block token params received from > NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec > 17/12/19 15:07:02 INFO block.BlockTokenSecretManager: Setting block keys > 17/12/19 15:07:02 INFO balancer.KeyManager: Update block keys every 2hrs, > 30mins, 0sec > 17/12/19 15:07:02 INFO command.Command: Reporting top -100 DataNode(s) > benefiting from running DiskBalancer. > Processing report command > Reporting top -100 DataNode(s) benefiting from running DiskBalancer. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12938) TestErasureCodigCLI testAll failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300418#comment-16300418
]
Ajay Kumar commented on HDFS-12938:
---
thanks [~eddyxu] for commit and [~xiaochen],[~shahrs87] for review.
> TestErasureCodigCLI testAll failing consistently.
> -
>
> Key: HDFS-12938
> URL: https://issues.apache.org/jira/browse/HDFS-12938
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-12938.001.patch
>
>
> {{TestErasureCodingCLI#testAll}} is failing consistently.
> It failed in this precommit:
> https://builds.apache.org/job/PreCommit-HDFS-Build/22435/testReport/org.apache.hadoop.cli/TestErasureCodingCLI/testAll/
> I ran locally on my laptop and it failed too.
> It failed with this stack trace:
> {noformat}
> java.lang.AssertionError: One of the tests failed. See the Detailed results
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:264)
> at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:126)
> at
> org.apache.hadoop.cli.TestErasureCodingCLI.tearDown(TestErasureCodingCLI.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Below is the detailed report from
> {{org.apache.hadoop.cli.TestErasureCodingCLI-output.txt}}.
> {noformat}
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(156)) -
> ---
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(157)) - Test ID: [15]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(158)) -Test Description:
> [setPolicy : set policy on non-empty directory]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(159)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -mkdir /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -touchz /ecdir/file1]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -setPolicy -policy RS-6-3-1024k -path /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(167)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(170)) -Cleanup Commands: [-fs
> hdfs://localhost:52345 -rm -R /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(174)) -
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(178)) - Comparator:
> [SubstringComparator]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(180)) - Comparision result:
> [fail]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(182)) - Expected output:
> [Warning: setting erasure coding policy on an non-empty directory will not
> automatically convert existing data to RS-6-3-1024]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(184)) - Actual output:
> [Set erasure coding policy RS-6-3-1024k on /ecdir
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically conve
[jira] [Updated] (HDFS-12860) StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB
[
https://issues.apache.org/jira/browse/HDFS-12860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12860:
-
Attachment: (was: HDFS-12860.00.patch)
> StripedBlockUtil#getRangesInternalBlocks throws exception for the block group
> size larger than 2GB
> --
>
> Key: HDFS-12860
> URL: https://issues.apache.org/jira/browse/HDFS-12860
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HDFS-12860.00.patch
>
>
> Running terasort on a cluster with 8 datanodes, 256GB data, using
> RS-3-2-1024k.
> The test data was generated by {{teragen}} with 32 mappers.
> The terasort benchmark fails with the following stack trace:
> {code}
> 17/11/27 14:44:31 INFO mapreduce.Job: map 45% reduce 0%
> 17/11/27 14:44:33 INFO mapreduce.Job: Task Id :
> attempt_1510080297865_0160_m_08_0, Status : FAILED
> Error: java.lang.IllegalArgumentException
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:72)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil$VerticalRange.(StripedBlockUtil.java:701)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.getRangesForInternalBlocks(StripedBlockUtil.java:442)
> at
> org.apache.hadoop.hdfs.util.StripedBlockUtil.divideOneStripe(StripedBlockUtil.java:311)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:308)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:562)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12938) TestErasureCodigCLI testAll failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-12938:
-
Resolution: Fixed
Fix Version/s: 3.0.1
3.1.0
Target Version/s: 3.0.0, 3.0.1
Status: Resolved (was: Patch Available)
Thanks for the quick response, [~ajayydv]! Committed to trunk and branch-3
> TestErasureCodigCLI testAll failing consistently.
> -
>
> Key: HDFS-12938
> URL: https://issues.apache.org/jira/browse/HDFS-12938
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-12938.001.patch
>
>
> {{TestErasureCodingCLI#testAll}} is failing consistently.
> It failed in this precommit:
> https://builds.apache.org/job/PreCommit-HDFS-Build/22435/testReport/org.apache.hadoop.cli/TestErasureCodingCLI/testAll/
> I ran locally on my laptop and it failed too.
> It failed with this stack trace:
> {noformat}
> java.lang.AssertionError: One of the tests failed. See the Detailed results
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:264)
> at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:126)
> at
> org.apache.hadoop.cli.TestErasureCodingCLI.tearDown(TestErasureCodingCLI.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Below is the detailed report from
> {{org.apache.hadoop.cli.TestErasureCodingCLI-output.txt}}.
> {noformat}
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(156)) -
> ---
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(157)) - Test ID: [15]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(158)) -Test Description:
> [setPolicy : set policy on non-empty directory]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(159)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -mkdir /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -touchz /ecdir/file1]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -setPolicy -policy RS-6-3-1024k -path /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(167)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(170)) -Cleanup Commands: [-fs
> hdfs://localhost:52345 -rm -R /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(174)) -
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(178)) - Comparator:
> [SubstringComparator]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(180)) - Comparision result:
> [fail]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(182)) - Expected output:
> [Warning: setting erasure coding policy on an non-empty directory will not
> automatically convert existing data to RS-6-3-1024]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(184)) - Actual output:
> [Set erasure codin
[jira] [Assigned] (HDFS-12953) XORRawDecoder.doDecode throws NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy reassigned HDFS-12953:
-
Assignee: Manoj Govindassamy (was: Lei (Eddy) Xu)
> XORRawDecoder.doDecode throws NullPointerException
> --
>
> Key: HDFS-12953
> URL: https://issues.apache.org/jira/browse/HDFS-12953
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Lei (Eddy) Xu
>Assignee: Manoj Govindassamy
>
> Thanks [~danielpol] report on HDFS-12860.
> {noformat}
> 17/11/30 04:19:55 INFO mapreduce.Job: map 0% reduce 0%
> 17/11/30 04:20:01 INFO mapreduce.Job: Task Id :
> attempt_1512036058655_0003_m_02_0, Status : FAILED
> Error: java.lang.NullPointerException
> at
> org.apache.hadoop.io.erasurecode.rawcoder.XORRawDecoder.doDecode(XORRawDecoder.java:83)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:106)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:423)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:382)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:318)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:391)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:813)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:563)
> at
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:794)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12931) Hive external table create fails because of failure to fetch block MD5 checksum
[
https://issues.apache.org/jira/browse/HDFS-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300402#comment-16300402
]
Ajay Kumar commented on HDFS-12931:
---
[~msingh], thanks for working on this. Are you planning to add test case to
handle encryption error?
> Hive external table create fails because of failure to fetch block MD5
> checksum
> ---
>
> Key: HDFS-12931
> URL: https://issues.apache.org/jira/browse/HDFS-12931
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Attachments: HDFS-12931.001.patch
>
>
> Hive external table create fails because of HDFS fails to fetch block MD5
> checksum.
> This happens because of the InvalidEncryptionKeyException.
> {code}
> org.apache.hadoop.hdfs.protocol.datatransfer.InvalidEncryptionKeyException:
> Can't re-compute encryption key for nonce, since the required block key
> (keyID=-1675775329) doesn't exist. Current key: 1496422662
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiatedCipherOption(DataTransferSaslUtil.java:417)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.doSaslHandshake(SaslDataTransferClient.java:478)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.getEncryptedStreams(SaslDataTransferClient.java:300)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.send(SaslDataTransferClient.java:241)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient.newSocketSend(SaslDataTransferClient.java:142)
> at org.apache.hadoop.hdfs.DFSClient.connectToDN(DFSClient.java:2450)
> at org.apache.hadoop.hdfs.DFSClient.getFileChecksum(DFSClient.java:2310)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1569)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$30.doCall(DistributedFileSystem.java:1565)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileChecksum(DistributedFileSystem.java:1577)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12751) Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close
[
https://issues.apache.org/jira/browse/HDFS-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300395#comment-16300395
]
Chen Liang commented on HDFS-12751:
---
Thanks for the clarification [~nandakumar131]. Filed HDFS-12958 as follow up.
> Ozone: SCM: update container allocated size to container db for all the open
> containers in ContainerStateManager#close
> --
>
> Key: HDFS-12751
> URL: https://issues.apache.org/jira/browse/HDFS-12751
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Chen Liang
> Fix For: HDFS-7240
>
> Attachments: HDFS-12751-HDFS-7240.001.patch,
> HDFS-12751-HDFS-7240.002.patch
>
>
> Container allocated size is maintained in memory by
> {{ContainerStateManager}}, this has to be updated in container db when we
> shutdown SCM. {{ContainerStateManager#close}} will be called during SCM
> shutdown, so updating allocated size for all the open containers should be
> done here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12958) Ozone: remove setAllocatedBytes method in ContainerInfo
Chen Liang created HDFS-12958:
-
Summary: Ozone: remove setAllocatedBytes method in ContainerInfo
Key: HDFS-12958
URL: https://issues.apache.org/jira/browse/HDFS-12958
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Chen Liang
Priority: Minor
We may want to remove {{setAllocatedBytes}} method from {{ContainerInfo}} and
we keep all fields of {{ContainerInfo}} immutable, such that client won't
accidentally change {{ContainerInfo}} and rely on the changed instance.
An alternative of having {{setAllocatedBytes}} is to always create a new
{{ContainerInfo}} instance whenever it needs to be changed.
This is based on [this
comment|https://issues.apache.org/jira/browse/HDFS-12751?focusedCommentId=16299750&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16299750]
from HDFS-12751.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300386#comment-16300386 ] Uma Maheswara Rao G commented on HDFS-10285: *MEETING NOTES* We had a meeting on 20 Dec 2017, mainly with the members who involved in the discussions in SPS JIRA Attendees: Anu, Eddy, ATM, Chris, Andrew, Vinay, Rakesh, Uma We had long discussions on the options available, specifically # Starting SPS within NN # Starting SPS as separate Service # Hybrid: SPS inside NN like RM and have fancier policies outside # Modularize SPS: SPS talks to NN via interface and make possibility of SPS to pull out easily while keeping SPS inside NN as one option. After all discussions (most of the points iterated from this JIRA comment arguments), for majority clusters starting SPS within NN may be sufficient, however, for larger clusters, it is reasonable argument to start SPS separately. One another motivation for start thinking to modularizing approach is, slowly we can bring other similar services from NN into outside SPS in future. *So, as a conclusion, we thought we should have both options (SPS within NN and SPS as service) available.* One should be able to start SPS inside NN, no maintenance burden, Others should be able to start SPS as independent Service as well. The current implementation of SPS should serve as internal service and after refactoring, respective necessary code can be added to serve as Independent service. Thank you, Chris, for proposing this approach and thanks others for agreeing to it. SPS should refactor to get clean interface between NN and SPS. Right now SPS, talks to NN protocol for SPS calls, keeping SPS as separate service options, it may be necessary to start SPS RPC with its own IP:port (within NN or outside), so clients can always talk to SPS on that port, irrespective of where its running. This will keep the API clean between these approaches. When we start SPS outside, we should have security and HA: probably we will handle this post merge High level tasks: # SPS refactoring to get into the modularized approach # Starting SPS service on its own port (for within NN/ outside) # Provide necessary plugin implementations to serve as independent service by not disturbing the provision to start inside NN. # Add necessary start/stop scripts for SPS (only for SPS outside NN). A separate document to be posted specifically for explaining the component interactions when started as a separate service. And Tasks will be prioritized for the merge. The current design doc holds good for internal SPS. Finally, based on user interests, they can prefer to start SPS within NN or outside NN @ attendees: please correct me if I miss any points covered in discussion. *[~daryn] : hope this is agreeable to you. Please feel free to comment if any concerns.* Thank you all for the productive discussions. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-10285-consolidated-merge-patch-02.patch, > HDFS-10285-consolidated-merge-patch-03.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf, > Storage-Policy-Satisfier-in-HDFS-Oct-26-2017.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename oper
[jira] [Commented] (HDFS-12957) Multi master name node
[ https://issues.apache.org/jira/browse/HDFS-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300377#comment-16300377 ] Anu Engineer commented on HDFS-12957: - [~as3525] There are several issues we need to solve to make a truly distributed namenode. Here are some other JIRAs links that talk about get the reads from all Namesnodes (that is all 3). Truly getting to a distributed Namenode is hard. * https://issues.apache.org/jira/browse/HDFS-12943 * https://docs.google.com/document/d/1WUwdFG09e7biEetWIN9a65j3Vpgp8rz50UmODM6PXLs/edit# * https://issues.apache.org/jira/browse/HDFS-6469 These documents explain various issues that need to solved to get to a distributed but still consistent reads and writes from Namenodes or simply get to Namenodes with more performance. > Multi master name node > -- > > Key: HDFS-12957 > URL: https://issues.apache.org/jira/browse/HDFS-12957 > Project: Hadoop HDFS > Issue Type: Wish > Components: namenode >Affects Versions: 3.0.0 > Environment: Multi node high availability with high throughput >Reporter: Andrew Schiefelbein >Priority: Minor > > While I do appreciate that the HA and federated setups do take care of some > of the issues running in a multi node distributed fault tolerant way I > believe that a singe name node is a bit of a pain point for me. > My wish, and your mileage may vary, is that here could be a name node on each > data node system so that each system could act as a gateway and worker for > the cluster. > In my tests right now I can push nearly 2 million records / second through a > single node instance, but when I bring up 4 more nodes it doesn't go 4x > faster as it has to retrieve from the name node and go through it as a > gateway to the cluster. For speed and resiliency if this could be spread out > among n number of nodes and put behind a load balancer it would be the ideal > solution for distributed resilient and high throughput shared storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12950) [oiv] ls will fail in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12950:
Summary: [oiv] ls will fail in secure cluster (was: oiv will fail in
secure cluster)
> [oiv] ls will fail in secure cluster
> -
>
> Key: HDFS-12950
> URL: https://issues.apache.org/jira/browse/HDFS-12950
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> if we execute ls, it will throw following.
> {noformat}
> hdfs dfs -ls webhdfs://127.0.0.1:5978/
> ls: Invalid value for webhdfs parameter "op"
> {noformat}
> When client is configured with security (i.e "hadoop.security.authentication=
> KERBEROS) ,
> then webhdfs will request getdelegation token which is not implemented and
> hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12947) Limit the number of Snapshots allowed to be created for a Snapshottable Directory
[ https://issues.apache.org/jira/browse/HDFS-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12947: --- Status: Patch Available (was: Open) > Limit the number of Snapshots allowed to be created for a Snapshottable > Directory > - > > Key: HDFS-12947 > URL: https://issues.apache.org/jira/browse/HDFS-12947 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12947.001.patch > > > Currently, A snapshottable directory is able to accommodate 65,536 snapshots. > In case a directory has large no of snapshots , deletion of any of the > earlier snapshots take a lot of time which might lead to namenode crash > (HDFS-11225). > This jira is introduced to limit the no of the snapshots under a > snapshottable directory to a reasonable value(say 10) which can be overriden. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12947) Limit the number of Snapshots allowed to be created for a Snapshottable Directory
[ https://issues.apache.org/jira/browse/HDFS-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-12947: --- Attachment: HDFS-12947.001.patch Patch v1 adds a config "dfs.namenode.snapshot.max.limit" which limits the maximum no of snapshots under a snapshottable directory, default value of which is 10. If the configuration value is set to a negative value, it will set the max limit to 65536 as earlier. > Limit the number of Snapshots allowed to be created for a Snapshottable > Directory > - > > Key: HDFS-12947 > URL: https://issues.apache.org/jira/browse/HDFS-12947 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee > Attachments: HDFS-12947.001.patch > > > Currently, A snapshottable directory is able to accommodate 65,536 snapshots. > In case a directory has large no of snapshots , deletion of any of the > earlier snapshots take a lot of time which might lead to namenode crash > (HDFS-11225). > This jira is introduced to limit the no of the snapshots under a > snapshottable directory to a reasonable value(say 10) which can be overriden. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12957) Multi master name node
[ https://issues.apache.org/jira/browse/HDFS-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300317#comment-16300317 ] Andrew Schiefelbein commented on HDFS-12957: Someone just pointed this out to me: https://community.hortonworks.com/articles/103176/hdfs-settings-for-better-hadoop-performance.html The short circuit reads will work for my local processes, but for distributed ones having the ability to have name nodes behind a load balancer would be a wonderful addition. > Multi master name node > -- > > Key: HDFS-12957 > URL: https://issues.apache.org/jira/browse/HDFS-12957 > Project: Hadoop HDFS > Issue Type: Wish > Components: namenode >Affects Versions: 3.0.0 > Environment: Multi node high availability with high throughput >Reporter: Andrew Schiefelbein >Priority: Minor > > While I do appreciate that the HA and federated setups do take care of some > of the issues running in a multi node distributed fault tolerant way I > believe that a singe name node is a bit of a pain point for me. > My wish, and your mileage may vary, is that here could be a name node on each > data node system so that each system could act as a gateway and worker for > the cluster. > In my tests right now I can push nearly 2 million records / second through a > single node instance, but when I bring up 4 more nodes it doesn't go 4x > faster as it has to retrieve from the name node and go through it as a > gateway to the cluster. For speed and resiliency if this could be spread out > among n number of nodes and put behind a load balancer it would be the ideal > solution for distributed resilient and high throughput shared storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12957) Multi master name node
[ https://issues.apache.org/jira/browse/HDFS-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schiefelbein updated HDFS-12957: --- Priority: Minor (was: Major) > Multi master name node > -- > > Key: HDFS-12957 > URL: https://issues.apache.org/jira/browse/HDFS-12957 > Project: Hadoop HDFS > Issue Type: Wish > Components: namenode >Affects Versions: 3.0.0 > Environment: Multi node high availability with high throughput >Reporter: Andrew Schiefelbein >Priority: Minor > > While I do appreciate that the HA and federated setups do take care of some > of the issues running in a multi node distributed fault tolerant way I > believe that a singe name node is a bit of a pain point for me. > My wish, and your mileage may vary, is that here could be a name node on each > data node system so that each system could act as a gateway and worker for > the cluster. > In my tests right now I can push nearly 2 million records / second through a > single node instance, but when I bring up 4 more nodes it doesn't go 4x > faster as it has to retrieve from the name node and go through it as a > gateway to the cluster. For speed and resiliency if this could be spread out > among n number of nodes and put behind a load balancer it would be the ideal > solution for distributed resilient and high throughput shared storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12950) oiv will fail in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300290#comment-16300290
]
Rushabh S Shah commented on HDFS-12950:
---
Thanks [~brahmareddy] for the documentation link.
I was unaware of this feature.
> oiv will fail in secure cluster
>
>
> Key: HDFS-12950
> URL: https://issues.apache.org/jira/browse/HDFS-12950
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> if we execute ls, it will throw following.
> {noformat}
> hdfs dfs -ls webhdfs://127.0.0.1:5978/
> ls: Invalid value for webhdfs parameter "op"
> {noformat}
> When client is configured with security (i.e "hadoop.security.authentication=
> KERBEROS) ,
> then webhdfs will request getdelegation token which is not implemented and
> hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12938) TestErasureCodigCLI testAll failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300282#comment-16300282
]
genericqa commented on HDFS-12938:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12938 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903087/HDFS-12938.001.patch |
| Optional Tests | asflicense unit xml |
| uname | Linux 45eb62e0e46a 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c8ff0cc |
| maven | version: Apache Maven 3.3.9 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/22485/testReport/ |
| Max. process+thread count | 313 (vs. ulimit of 5000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/22485/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestErasureCodigCLI testAll failing consistently.
> -
>
> Key: HDFS-12938
> URL: https://issues.apache.org/jira/browse/HDFS-12938
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
> Attachments: HDFS-12938.001.patch
>
>
> {{TestErasureCodingCLI#testAll}} is failing consistently.
> It failed in this precommit:
> https://builds.apache.org/job/PreCommit-HDFS-Build/22435/testReport/org.apache.hadoop.cli/TestErasureCodingCLI/testAll/
> I ran locally on my laptop and it failed too.
> It failed with this stack trace:
> {noformat}
> java.lang.AssertionError: One of the tests failed. See the Detailed results
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:264)
> at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:126)
> at
> org.apache.hadoop.cli.TestErasureCodingCLI.tearDown(TestErasureCodingCLI.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runne
[jira] [Updated] (HDFS-12950) oiv will fail in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-12950:
Affects Version/s: 2.7.0
> oiv will fail in secure cluster
>
>
> Key: HDFS-12950
> URL: https://issues.apache.org/jira/browse/HDFS-12950
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> if we execute ls, it will throw following.
> {noformat}
> hdfs dfs -ls webhdfs://127.0.0.1:5978/
> ls: Invalid value for webhdfs parameter "op"
> {noformat}
> When client is configured with security (i.e "hadoop.security.authentication=
> KERBEROS) ,
> then webhdfs will request getdelegation token which is not implemented and
> hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12950) oiv will fail in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300273#comment-16300273
]
Brahma Reddy Battula commented on HDFS-12950:
-
bq.This jira is related to oiv and in the example you are issuing ls command.
when we execute {{"hdfs oiv -i fsimage"}}, It launches a HTTP server that
exposes read-only WebHDFS API.Here default port is {{5978}}.So I used same to
list the files. Please refer example
[here|http://hadoop.apache.org/docs/r2.9.0/hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html]
for more.
bq.I tried the same command on 2.8 cluster and I am getting some different
error.
looks with {{5978}} nothing is running,so it's retrying.
bq.Also can you mention the "Affects Version" ?
It's applicable to {{2.6}} + all the versions,will update same..
> oiv will fail in secure cluster
>
>
> Key: HDFS-12950
> URL: https://issues.apache.org/jira/browse/HDFS-12950
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> if we execute ls, it will throw following.
> {noformat}
> hdfs dfs -ls webhdfs://127.0.0.1:5978/
> ls: Invalid value for webhdfs parameter "op"
> {noformat}
> When client is configured with security (i.e "hadoop.security.authentication=
> KERBEROS) ,
> then webhdfs will request getdelegation token which is not implemented and
> hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12957) Multi master name node
Andrew Schiefelbein created HDFS-12957: -- Summary: Multi master name node Key: HDFS-12957 URL: https://issues.apache.org/jira/browse/HDFS-12957 Project: Hadoop HDFS Issue Type: Wish Components: namenode Affects Versions: 3.0.0 Environment: Multi node high availability with high throughput Reporter: Andrew Schiefelbein While I do appreciate that the HA and federated setups do take care of some of the issues running in a multi node distributed fault tolerant way I believe that a singe name node is a bit of a pain point for me. My wish, and your mileage may vary, is that here could be a name node on each data node system so that each system could act as a gateway and worker for the cluster. In my tests right now I can push nearly 2 million records / second through a single node instance, but when I bring up 4 more nodes it doesn't go 4x faster as it has to retrieve from the name node and go through it as a gateway to the cluster. For speed and resiliency if this could be spread out among n number of nodes and put behind a load balancer it would be the ideal solution for distributed resilient and high throughput shared storage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12938) TestErasureCodigCLI testAll failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300205#comment-16300205
]
Rushabh S Shah edited comment on HDFS-12938 at 12/21/17 4:33 PM:
-
Surprised to know that the test code is so fragile.
We are comparing the whole string including "a or an".
Hopefully we can come up with better test cases in future.
was (Author: shahrs87):
Surprised to know that the code is so fragile.
We are comparing the whole string including "a or an".
Hopefully we can come up with better test cases in future.
> TestErasureCodigCLI testAll failing consistently.
> -
>
> Key: HDFS-12938
> URL: https://issues.apache.org/jira/browse/HDFS-12938
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
> Attachments: HDFS-12938.001.patch
>
>
> {{TestErasureCodingCLI#testAll}} is failing consistently.
> It failed in this precommit:
> https://builds.apache.org/job/PreCommit-HDFS-Build/22435/testReport/org.apache.hadoop.cli/TestErasureCodingCLI/testAll/
> I ran locally on my laptop and it failed too.
> It failed with this stack trace:
> {noformat}
> java.lang.AssertionError: One of the tests failed. See the Detailed results
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:264)
> at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:126)
> at
> org.apache.hadoop.cli.TestErasureCodingCLI.tearDown(TestErasureCodingCLI.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Below is the detailed report from
> {{org.apache.hadoop.cli.TestErasureCodingCLI-output.txt}}.
> {noformat}
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(156)) -
> ---
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(157)) - Test ID: [15]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(158)) -Test Description:
> [setPolicy : set policy on non-empty directory]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(159)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -mkdir /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -touchz /ecdir/file1]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -setPolicy -policy RS-6-3-1024k -path /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(167)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(170)) -Cleanup Commands: [-fs
> hdfs://localhost:52345 -rm -R /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(174)) -
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(178)) - Comparator:
> [SubstringComparator]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(180)) - Comparision result:
> [fail]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(182)) - Expected output:
> [Warning: setting erasure coding policy on an non-empty directory will not
> automatically convert existing data
[jira] [Commented] (HDFS-12938) TestErasureCodigCLI testAll failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300205#comment-16300205
]
Rushabh S Shah commented on HDFS-12938:
---
Surprised to know that the code is so fragile.
We are comparing the whole string including "a or an".
Hopefully we can come up with better test cases in future.
> TestErasureCodigCLI testAll failing consistently.
> -
>
> Key: HDFS-12938
> URL: https://issues.apache.org/jira/browse/HDFS-12938
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
> Attachments: HDFS-12938.001.patch
>
>
> {{TestErasureCodingCLI#testAll}} is failing consistently.
> It failed in this precommit:
> https://builds.apache.org/job/PreCommit-HDFS-Build/22435/testReport/org.apache.hadoop.cli/TestErasureCodingCLI/testAll/
> I ran locally on my laptop and it failed too.
> It failed with this stack trace:
> {noformat}
> java.lang.AssertionError: One of the tests failed. See the Detailed results
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:264)
> at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:126)
> at
> org.apache.hadoop.cli.TestErasureCodingCLI.tearDown(TestErasureCodingCLI.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Below is the detailed report from
> {{org.apache.hadoop.cli.TestErasureCodingCLI-output.txt}}.
> {noformat}
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(156)) -
> ---
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(157)) - Test ID: [15]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(158)) -Test Description:
> [setPolicy : set policy on non-empty directory]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(159)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -mkdir /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -touchz /ecdir/file1]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -setPolicy -policy RS-6-3-1024k -path /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(167)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(170)) -Cleanup Commands: [-fs
> hdfs://localhost:52345 -rm -R /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(174)) -
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(178)) - Comparator:
> [SubstringComparator]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(180)) - Comparision result:
> [fail]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(182)) - Expected output:
> [Warning: setting erasure coding policy on an non-empty directory will not
> automatically convert existing data to RS-6-3-1024]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(184)) - Actual output:
> [Set erasure coding policy RS-6-3-1024k on /ecdir
> Warning: setting erasure coding p
[jira] [Updated] (HDFS-12938) TestErasureCodigCLI testAll failing consistently.
[
https://issues.apache.org/jira/browse/HDFS-12938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-12938:
--
Status: Patch Available (was: Open)
> TestErasureCodigCLI testAll failing consistently.
> -
>
> Key: HDFS-12938
> URL: https://issues.apache.org/jira/browse/HDFS-12938
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, hdfs
>Affects Versions: 3.1.0
>Reporter: Rushabh S Shah
>Assignee: Ajay Kumar
> Attachments: HDFS-12938.001.patch
>
>
> {{TestErasureCodingCLI#testAll}} is failing consistently.
> It failed in this precommit:
> https://builds.apache.org/job/PreCommit-HDFS-Build/22435/testReport/org.apache.hadoop.cli/TestErasureCodingCLI/testAll/
> I ran locally on my laptop and it failed too.
> It failed with this stack trace:
> {noformat}
> java.lang.AssertionError: One of the tests failed. See the Detailed results
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:264)
> at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:126)
> at
> org.apache.hadoop.cli.TestErasureCodingCLI.tearDown(TestErasureCodingCLI.java:77)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Below is the detailed report from
> {{org.apache.hadoop.cli.TestErasureCodingCLI-output.txt}}.
> {noformat}
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(156)) -
> ---
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(157)) - Test ID: [15]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(158)) -Test Description:
> [setPolicy : set policy on non-empty directory]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(159)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -mkdir /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -touchz /ecdir/file1]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(163)) - Test Commands: [-fs
> hdfs://localhost:52345 -setPolicy -policy RS-6-3-1024k -path /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(167)) -
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(170)) -Cleanup Commands: [-fs
> hdfs://localhost:52345 -rm -R /ecdir]
> 2017-12-18 09:25:44,821 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(174)) -
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(178)) - Comparator:
> [SubstringComparator]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(180)) - Comparision result:
> [fail]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(182)) - Expected output:
> [Warning: setting erasure coding policy on an non-empty directory will not
> automatically convert existing data to RS-6-3-1024]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:displayResults(184)) - Actual output:
> [Set erasure coding policy RS-6-3-1024k on /ecdir
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> ]
> 2017-12-18 09:25:44,822 [Thread-0] INFO cli.CLITestHelper
> (CLITestHelper.java:di
[jira] [Commented] (HDFS-12950) oiv will fail in secure cluster
[
https://issues.apache.org/jira/browse/HDFS-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300190#comment-16300190
]
Rushabh S Shah commented on HDFS-12950:
---
bq. hdfs dfs -ls webhdfs://127.0.0.1:5978/
This jira is related to oiv and in the example you are issuing {{ls}} command.
I tried the same command on 2.8 cluster and I am getting some different error.
{noformat}
hdfs dfs -ls webhdfs://:5978/
17/12/21 15:39:35 INFO web.WebHdfsFileSystem: Retrying connect to
namenode::5978. Already retried 0 time(s); retry policy is , delay 0ms.
ls: :5978: Unexpected HTTP response: code=404 != 200,
op=GETFILESTATUS, message=Not Found
{noformat}
I am sure I am missing something.
Also can you mention the "Affects Version" ?
> oiv will fail in secure cluster
>
>
> Key: HDFS-12950
> URL: https://issues.apache.org/jira/browse/HDFS-12950
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
>
> if we execute ls, it will throw following.
> {noformat}
> hdfs dfs -ls webhdfs://127.0.0.1:5978/
> ls: Invalid value for webhdfs parameter "op"
> {noformat}
> When client is configured with security (i.e "hadoop.security.authentication=
> KERBEROS) ,
> then webhdfs will request getdelegation token which is not implemented and
> hence it will throw “ls: Invalid value for webhdfs parameter "op"”.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12812) Ozone: dozone: initialize scm and ksm directory on cluster startup
[ https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300031#comment-16300031 ] Elek, Marton commented on HDFS-12812: - [~anu] Thx to test it. You are right, the image was not built on the dockerhup. I tested with locally created image and didn't notice it. Could you please try it again? I tested it with a new machine, it should work after docker-compose pull. [~nandakumar131] Thx, you are right. I have already added this to the script in the last patch but didn't modify the title. Now I fixed the jira. The patch handles both scm and ksm changes. > Ozone: dozone: initialize scm and ksm directory on cluster startup > -- > > Key: HDFS-12812 > URL: https://issues.apache.org/jira/browse/HDFS-12812 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HADOOP-12812-HDFS-7240.001.patch, > HDFS-12812-HDFS-7240.002.patch > > > HDFS-12739 fixed the scm, but after the patch it couldn't be started without > a separated `hdfs scm -init` any more. Unfortunatelly it breaks the > docker-compose functionality. > This is handled int the starter script of the base runner docker image for > namenode. I also fixed this in the runner docker image > (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7) > will upload the improved patch to the HADOOP-14898. > In this patch I just add a new environment variable to init the scm if the > version file doesn't exist. > UPDATE: the patch also contains envrionment variable to initialize the ksm. > To test: > Do a full build and go to the dev-support/compose/ozone: > ``` > docker-compose pull > docker-compose up -d > ``` > Note: the pull is important as the new docker image -- which could handle the > new environment variable -- should be used -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12812) Ozone: dozone: initialize scm and ksm directory on cluster startup
[ https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-12812: Description: HDFS-12739 fixed the scm, but after the patch it couldn't be started without a separated `hdfs scm -init` any more. Unfortunatelly it breaks the docker-compose functionality. This is handled int the starter script of the base runner docker image for namenode. I also fixed this in the runner docker image (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7) will upload the improved patch to the HADOOP-14898. In this patch I just add a new environment variable to init the scm if the version file doesn't exist. UPDATE: the patch also contains envrionment variable to initialize the ksm. To test: Do a full build and go to the dev-support/compose/ozone: ``` docker-compose pull docker-compose up -d ``` Note: the pull is important as the new docker image -- which could handle the new environment variable -- should be used was: HDFS-12739 fixed the scm, but after the patch it couldn't be started without a separated `hdfs scm -init` any more. Unfortunatelly it breaks the docker-compose functionality. This is handled int the starter script of the base runner docker image for namenode. I also fixed this in the runner docker image (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7) will upload the improved patch to the HADOOP-14898. In this patch I just add a new environment variable to init the scm if the version file doesn't exist. To test: Do a full build and go to the dev-support/compose/ozone: ``` docker-compose pull docker-compose up -d ``` Note: the pull is important as the new docker image -- which could handle the new environment variable -- should be used > Ozone: dozone: initialize scm and ksm directory on cluster startup > -- > > Key: HDFS-12812 > URL: https://issues.apache.org/jira/browse/HDFS-12812 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HADOOP-12812-HDFS-7240.001.patch, > HDFS-12812-HDFS-7240.002.patch > > > HDFS-12739 fixed the scm, but after the patch it couldn't be started without > a separated `hdfs scm -init` any more. Unfortunatelly it breaks the > docker-compose functionality. > This is handled int the starter script of the base runner docker image for > namenode. I also fixed this in the runner docker image > (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7) > will upload the improved patch to the HADOOP-14898. > In this patch I just add a new environment variable to init the scm if the > version file doesn't exist. > UPDATE: the patch also contains envrionment variable to initialize the ksm. > To test: > Do a full build and go to the dev-support/compose/ozone: > ``` > docker-compose pull > docker-compose up -d > ``` > Note: the pull is important as the new docker image -- which could handle the > new environment variable -- should be used -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12812) Ozone: dozone: initialize scm and ksm directory on cluster startup
[ https://issues.apache.org/jira/browse/HDFS-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-12812: Summary: Ozone: dozone: initialize scm and ksm directory on cluster startup (was: Ozone: dozone: initialize scm directory on cluster startup) > Ozone: dozone: initialize scm and ksm directory on cluster startup > -- > > Key: HDFS-12812 > URL: https://issues.apache.org/jira/browse/HDFS-12812 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HADOOP-12812-HDFS-7240.001.patch, > HDFS-12812-HDFS-7240.002.patch > > > HDFS-12739 fixed the scm, but after the patch it couldn't be started without > a separated `hdfs scm -init` any more. Unfortunatelly it breaks the > docker-compose functionality. > This is handled int the starter script of the base runner docker image for > namenode. I also fixed this in the runner docker image > (https://github.com/elek/hadoop/commit/b347eb4bfca37d84dbcdd8c4bf353219d876a9b7) > will upload the improved patch to the HADOOP-14898. > In this patch I just add a new environment variable to init the scm if the > version file doesn't exist. > To test: > Do a full build and go to the dev-support/compose/ozone: > ``` > docker-compose pull > docker-compose up -d > ``` > Note: the pull is important as the new docker image -- which could handle the > new environment variable -- should be used -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12908) Ozone: write chunk call fails because of Metrics registry exception
[
https://issues.apache.org/jira/browse/HDFS-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1639#comment-1639
]
Nanda kumar commented on HDFS-12908:
I have committed this to the feature branch, thanks [~msingh] for the
contribution.
> Ozone: write chunk call fails because of Metrics registry exception
> ---
>
> Key: HDFS-12908
> URL: https://issues.apache.org/jira/browse/HDFS-12908
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12908-HDFS-7240.001.patch,
> HDFS-12908-HDFS-7240.002.patch, HDFS-12908-HDFS-7240.003.patch,
> HDFS-12908-HDFS-7240.004.patch
>
>
> write chunk call fail because of Metric registration exception.
> {code}
> 2017-12-08 04:02:19,894 WARN org.apache.hadoop.metrics2.util.MBeans: Error
> creating MBean object name:
> Hadoop:service=Ozone,name=RocksDbStore,dbName=container.db
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=Ozone,name=RocksDbStore,dbName=container.db already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:135)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:110)
> at
> org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:155)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:87)
> at org.apache.hadoop.utils.RocksDBStore.(RocksDBStore.java:77)
> at
> org.apache.hadoop.utils.MetadataStoreBuilder.build(MetadataStoreBuilder.java:115)
> at
> org.apache.hadoop.ozone.container.common.utils.ContainerCache.getDB(ContainerCache.java:138)
> at
> org.apache.hadoop.ozone.container.common.helpers.KeyUtils.getDB(KeyUtils.java:65)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.readContainerInfo(ContainerManagerImpl.java:261)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.createContainer(ContainerManagerImpl.java:330)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.handleCreateContainer(Dispatcher.java:399)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.containerProcessHandler(Dispatcher.java:158)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.dispatch(Dispatcher.java:105)
> at
> org.apache.hadoop.ozone.container.common.transport.server.XceiverServerHandler.channelRead0(XceiverServerHandler.java:61)
> at
> org.apache.hadoop.ozone.container.common.transport.server.XceiverServerHandler.channelRead0(XceiverServerHandler.java:32)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultCha
[jira] [Updated] (HDFS-12908) Ozone: write chunk call fails because of Metrics registry exception
[
https://issues.apache.org/jira/browse/HDFS-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDFS-12908:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Ozone: write chunk call fails because of Metrics registry exception
> ---
>
> Key: HDFS-12908
> URL: https://issues.apache.org/jira/browse/HDFS-12908
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12908-HDFS-7240.001.patch,
> HDFS-12908-HDFS-7240.002.patch, HDFS-12908-HDFS-7240.003.patch,
> HDFS-12908-HDFS-7240.004.patch
>
>
> write chunk call fail because of Metric registration exception.
> {code}
> 2017-12-08 04:02:19,894 WARN org.apache.hadoop.metrics2.util.MBeans: Error
> creating MBean object name:
> Hadoop:service=Ozone,name=RocksDbStore,dbName=container.db
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=Ozone,name=RocksDbStore,dbName=container.db already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:135)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:110)
> at
> org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:155)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:87)
> at org.apache.hadoop.utils.RocksDBStore.(RocksDBStore.java:77)
> at
> org.apache.hadoop.utils.MetadataStoreBuilder.build(MetadataStoreBuilder.java:115)
> at
> org.apache.hadoop.ozone.container.common.utils.ContainerCache.getDB(ContainerCache.java:138)
> at
> org.apache.hadoop.ozone.container.common.helpers.KeyUtils.getDB(KeyUtils.java:65)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.readContainerInfo(ContainerManagerImpl.java:261)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.createContainer(ContainerManagerImpl.java:330)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.handleCreateContainer(Dispatcher.java:399)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.containerProcessHandler(Dispatcher.java:158)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.dispatch(Dispatcher.java:105)
> at
> org.apache.hadoop.ozone.container.common.transport.server.XceiverServerHandler.channelRead0(XceiverServerHandler.java:61)
> at
> org.apache.hadoop.ozone.container.common.transport.server.XceiverServerHandler.channelRead0(XceiverServerHandler.java:32)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io
[jira] [Commented] (HDFS-12908) Ozone: write chunk call fails because of Metrics registry exception
[
https://issues.apache.org/jira/browse/HDFS-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1630#comment-1630
]
Nanda kumar commented on HDFS-12908:
I will commit this shortly.
> Ozone: write chunk call fails because of Metrics registry exception
> ---
>
> Key: HDFS-12908
> URL: https://issues.apache.org/jira/browse/HDFS-12908
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12908-HDFS-7240.001.patch,
> HDFS-12908-HDFS-7240.002.patch, HDFS-12908-HDFS-7240.003.patch,
> HDFS-12908-HDFS-7240.004.patch
>
>
> write chunk call fail because of Metric registration exception.
> {code}
> 2017-12-08 04:02:19,894 WARN org.apache.hadoop.metrics2.util.MBeans: Error
> creating MBean object name:
> Hadoop:service=Ozone,name=RocksDbStore,dbName=container.db
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=Ozone,name=RocksDbStore,dbName=container.db already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:135)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:110)
> at
> org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:155)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:87)
> at org.apache.hadoop.utils.RocksDBStore.(RocksDBStore.java:77)
> at
> org.apache.hadoop.utils.MetadataStoreBuilder.build(MetadataStoreBuilder.java:115)
> at
> org.apache.hadoop.ozone.container.common.utils.ContainerCache.getDB(ContainerCache.java:138)
> at
> org.apache.hadoop.ozone.container.common.helpers.KeyUtils.getDB(KeyUtils.java:65)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.readContainerInfo(ContainerManagerImpl.java:261)
> at
> org.apache.hadoop.ozone.container.common.impl.ContainerManagerImpl.createContainer(ContainerManagerImpl.java:330)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.handleCreateContainer(Dispatcher.java:399)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.containerProcessHandler(Dispatcher.java:158)
> at
> org.apache.hadoop.ozone.container.common.impl.Dispatcher.dispatch(Dispatcher.java:105)
> at
> org.apache.hadoop.ozone.container.common.transport.server.XceiverServerHandler.channelRead0(XceiverServerHandler.java:61)
> at
> org.apache.hadoop.ozone.container.common.transport.server.XceiverServerHandler.channelRead0(XceiverServerHandler.java:32)
> at
> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1302)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io.netty.channel.n
[jira] [Commented] (HDFS-12940) Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally
[
https://issues.apache.org/jira/browse/HDFS-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1629#comment-1629
]
Nanda kumar commented on HDFS-12940:
Even with {{OZONE_OPEN_KEY_CLEANUP_SERVICE_INTERVAL_SECONDS = 3}} and
{{OZONE_OPEN_KEY_EXPIRE_THRESHOLD_SECONDS = 2}} few test-cases are failing with
{{java.io.IOException: Commit key failed, error:KEY_NOT_FOUND}}
{noformat}
java.io.IOException: Commit key failed, error:KEY_NOT_FOUND
at
org.apache.hadoop.ozone.ksm.protocolPB.KeySpaceManagerProtocolClientSideTranslatorPB.commitKey(KeySpaceManagerProtocolClientSideTranslatorPB.java:598)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:309)
at
org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:58)
at
org.apache.hadoop.ozone.ksm.TestKeySpaceManager.testWriteSize(TestKeySpaceManager.java:1061)
...
---
java.io.IOException: Commit key failed, error:KEY_NOT_FOUND
at
org.apache.hadoop.ozone.ksm.protocolPB.KeySpaceManagerProtocolClientSideTranslatorPB.commitKey(KeySpaceManagerProtocolClientSideTranslatorPB.java:598)
at
org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.close(ChunkGroupOutputStream.java:309)
at
org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:58)
at
org.apache.hadoop.ozone.ksm.TestKeySpaceManager.testListKeys(TestKeySpaceManager.java:814)
...
{noformat}
> Ozone: KSM: TestKeySpaceManager#testExpiredOpenKey fails occasionally
> -
>
> Key: HDFS-12940
> URL: https://issues.apache.org/jira/browse/HDFS-12940
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Nanda kumar
>Assignee: Nanda kumar
> Attachments: HDFS-12940-HDFS-7240.000.patch,
> HDFS-12940-HDFS-7240.001.patch
>
>
> {{TestKeySpaceManager#testExpiredOpenKey}} is flaky.
> In {{testExpiredOpenKey}} we are opening four keys for writing and wait for
> them to expire (without committing). Verification/Assertion is done by
> querying {{MiniOzoneCluster}} and matching the count. Since the {{cluster}}
> instance of {{MiniOzoneCluster}} is shared between test-cases in
> {{TestKeySpaceManager}}, we should not rely on the count. The verification
> should only happen by matching the keyNames and not with the count.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
