[jira] [Commented] (HDDS-347) TestCloseContainerByPipeline#testCloseContainerViaStandaAlone sometimes fails
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582000#comment-16582000
]
LiXin Ge commented on HDDS-347:
---
Thanks [~xyao] for reviewing and the susgestions.
> TestCloseContainerByPipeline#testCloseContainerViaStandaAlone sometimes fails
> -
>
> Key: HDDS-347
> URL: https://issues.apache.org/jira/browse/HDDS-347
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-347.000.patch, HDDS-347.001.patch
>
>
> This issue was finded in the automatic JenKins unit test of HDDS-265.
> The container life cycle state is : Open -> Closing -> closed, this test
> submit the container close command and wait for container state change to
> *not equal to open*, actually even when the state condition(not equal to
> open) is satisfied, the container may still in process of closing, so the LOG
> which will printf after the container closed can't be find sometimes and the
> test fails.
> {code:java|title=KeyValueContainer.java|borderStyle=solid}
> try {
> writeLock();
> containerData.closeContainer();
> File containerFile = getContainerFile();
> // update the new container data to .container File
> updateContainerFile(containerFile);
> } catch (StorageContainerException ex) {
> {code}
> Looking at the code above, the container state changes from CLOSING to CLOSED
> in the first step, the remaining *updateContainerFile* may take hundreds of
> milliseconds, so even we modify the test logic to wait for the *CLOSED* state
> will not guarantee the test success, too.
> These are two way to fix this:
> 1, Remove one of the double check which depends on the LOG.
> 2, If we have to preserve the double check, we should wait for the *CLOSED*
> state and sleep for a while to wait for the LOG appears.
> patch 000 is based on the second way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-179) CloseContainer command should be executed only if all the prior "Write" type container requests get executed
[ https://issues.apache.org/jira/browse/HDDS-179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581998#comment-16581998 ] Mukul Kumar Singh commented on HDDS-179: Thanks for updating the patch [~shashikant]. The patch looks excellent to me. Some very minor comments. I am +1 once these issues are fixed. 1) ContainerStateMachine.java#getCommitChunkFutureMap, the function always returns null. Also I see that we are exporting statemachine map for testing, I feel these functions can be moved to the test file. > CloseContainer command should be executed only if all the prior "Write" type > container requests get executed > - > > Key: HDDS-179 > URL: https://issues.apache.org/jira/browse/HDDS-179 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client, Ozone Datanode >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 0.2.1 > > Attachments: HDDS-179.01.patch, HDDS-179.02.patch, HDDS-179.03.patch, > HDDS-179.04.patch, HDDS-179.05.patch, HDDS-179.06.patch, HDDS-179.07.patch, > HDDS-179.08.patch, HDDS-179.09,patch, HDDS-179.10.patch > > > When a close Container command request comes to a Datanode (via SCM hearbeat > response) through the Ratis protocol, all the prior enqueued "Write" type of > request like WriteChunk etc should be executed first before CloseContainer > request gets executed. This synchronization needs to be handled in the > containerStateMachine. This Jira aims to address this. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13829) Needn't get min value of scan index and length for blockpoolReport
[
https://issues.apache.org/jira/browse/HDFS-13829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin updated HDFS-13829:
--
Status: Patch Available (was: Open)
> Needn't get min value of scan index and length for blockpoolReport
> --
>
> Key: HDFS-13829
> URL: https://issues.apache.org/jira/browse/HDFS-13829
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
> Environment:
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13829.001.patch
>
>
> When i read the scan() of DirectoryScanner class in datanode, i found the
> following condition code could be more simplify and easy to understand.
> DirectoryScanner.java
> {code:java}
> if (d < blockpoolReport.length) {
> // There may be multiple on-disk records for the same block,
> don't increment
> // the memory record pointer if so.
> ScanInfo nextInfo = blockpoolReport[Math.min(d,
> blockpoolReport.length - 1)];
> if (nextInfo.getBlockId() != info.getBlockId()) {
> ++m;
> }
> } else {
> ++m;
> }
> {code}
> as described above code segmet, i find the code of d < blockpoolReport.length
> and the max of d is blockpoolReport.length-1, so that result of Math.min(d,
> blockpoolReport.length - 1) always is d, so i think needn't to get min value
> of scan index and length for blockpoolReport.
> thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13829) Needn't get min value of scan index and length for blockpoolReport
[
https://issues.apache.org/jira/browse/HDFS-13829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin updated HDFS-13829:
--
Attachment: HDFS-13829.001.patch
> Needn't get min value of scan index and length for blockpoolReport
> --
>
> Key: HDFS-13829
> URL: https://issues.apache.org/jira/browse/HDFS-13829
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
> Environment:
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13829.001.patch
>
>
> When i read the scan() of DirectoryScanner class in datanode, i found the
> following condition code could be more simplify and easy to understand.
> DirectoryScanner.java
> {code:java}
> if (d < blockpoolReport.length) {
> // There may be multiple on-disk records for the same block,
> don't increment
> // the memory record pointer if so.
> ScanInfo nextInfo = blockpoolReport[Math.min(d,
> blockpoolReport.length - 1)];
> if (nextInfo.getBlockId() != info.getBlockId()) {
> ++m;
> }
> } else {
> ++m;
> }
> {code}
> as described above code segmet, i find the code of d < blockpoolReport.length
> and the max of d is blockpoolReport.length-1, so that result of Math.min(d,
> blockpoolReport.length - 1) always is d, so i think needn't to get min value
> of scan index and length for blockpoolReport.
> thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-350) ContainerMapping#flushContainerInfo doesn't set containerId
[ https://issues.apache.org/jira/browse/HDDS-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-350: Status: Patch Available (was: Open) > ContainerMapping#flushContainerInfo doesn't set containerId > --- > > Key: HDDS-350 > URL: https://issues.apache.org/jira/browse/HDDS-350 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-350.00.patch > > > ContainerMapping#flushContainerInfo doesn't set containerId which results in > containerId being null in flushed containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8277) Safemode enter fails when Standby NameNode is down
[
https://issues.apache.org/jira/browse/HDFS-8277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581991#comment-16581991
]
genericqa commented on HDFS-8277:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.client.impl.TestBlockReaderLocal |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-8277 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935659/HDFS-8277_5.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 24c293498070 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7dc79a8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24787/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24787/testReport/ |
| Max. process+thread count | 3068 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24787/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message
[jira] [Updated] (HDDS-350) ContainerMapping#flushContainerInfo doesn't set containerId
[ https://issues.apache.org/jira/browse/HDDS-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-350: Attachment: HDDS-350.00.patch > ContainerMapping#flushContainerInfo doesn't set containerId > --- > > Key: HDDS-350 > URL: https://issues.apache.org/jira/browse/HDDS-350 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-350.00.patch > > > ContainerMapping#flushContainerInfo doesn't set containerId which results in > containerId being null in flushed containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-350) ContainerMapping#flushContainerInfo doesn't set containerId
[ https://issues.apache.org/jira/browse/HDDS-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-350: Attachment: (was: HDDS-350.00.patch) > ContainerMapping#flushContainerInfo doesn't set containerId > --- > > Key: HDDS-350 > URL: https://issues.apache.org/jira/browse/HDDS-350 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > ContainerMapping#flushContainerInfo doesn't set containerId which results in > containerId being null in flushed containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-350) ContainerMapping#flushContainerInfo doesn't set containerId
[ https://issues.apache.org/jira/browse/HDDS-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-350: Attachment: HDDS-350.00.patch > ContainerMapping#flushContainerInfo doesn't set containerId > --- > > Key: HDDS-350 > URL: https://issues.apache.org/jira/browse/HDDS-350 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-350.00.patch > > > ContainerMapping#flushContainerInfo doesn't set containerId which results in > containerId being null in flushed containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13772) Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled
[ https://issues.apache.org/jira/browse/HDFS-13772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581921#comment-16581921 ] Ayush Saxena commented on HDFS-13772: - The method signature change is from void to boolean. After changing I don't think there would be any compilation issue with previous usage, as previously method was not returning anything. For the old versions it would be working as is but returned value will not be used. Do you think this is incompatible change? > Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling > Erasure coding policies which are already enabled/disabled > -- > > Key: HDFS-13772 > URL: https://issues.apache.org/jira/browse/HDFS-13772 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding >Affects Versions: 3.0.0 > Environment: 3 Node SuSE Linux cluster >Reporter: Souryakanta Dwivedy >Assignee: Ayush Saxena >Priority: Trivial > Attachments: EC_capture1.PNG, HDFS-13772-01.patch > > > Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding > policies which are already enabled/disabled > - Enable any Erasure coding policy like "RS-LEGACY-6-3-1024k" > - Check the console log display as "Erasure coding policy RS-LEGACY-6-3-1024k > is enabled" > - Again try to enable the same policy multiple times "hdfs ec -enablePolicy > -policy RS-LEGACY-6-3-1024k" > instead of throwing error message as ""policy already enabled"" it will > display same messages as "Erasure coding policy RS-LEGACY-6-3-1024k is > enabled" > - Also in NameNode log policy enabled logs are displaying multiple times > unnecessarily even though the policy is already enabled. > like this : 2018-07-27 18:50:35,084 INFO > org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager: Disable > the erasure coding policy RS-10-4-1024k > 2018-07-27 18:50:35,084 INFO > org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager: Disable > the erasure coding policy RS-10-4-1024k > 2018-07-27 18:50:35,084 INFO > org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager: Disable > the erasure coding policy RS-10-4-1024k > 2018-07-27 18:50:35,084 INFO > org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager: Enable the > erasure coding policy RS-LEGACY-6-3-1024k > 2018-07-27 18:50:35,084 INFO > org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager: Enable the > erasure coding policy RS-LEGACY-6-3-1024k > 2018-07-27 18:50:35,084 INFO > org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager: Enable the > erasure coding policy RS-LEGACY-6-3-1024k > - While executing the Erasure coding policy disable command also same type of > logs coming multiple times even though the policy is already > disabled.It should throw error message as ""policy is already disabled"" for > already disabled policy. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13655) RBF: Add missing ClientProtocol APIs to RBF
[
https://issues.apache.org/jira/browse/HDFS-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581899#comment-16581899
]
Yiqun Lin commented on HDFS-13655:
--
Hi [~xiaochen] and [~elgoiri], cut a new branch to implement remaining APIs
will be a good way. Can we create a branch as soon as possible? Then we can
speed up the work of this. We can still test and do some adjustment before the
final branch merge work.
I didn't have some experiences of cutting branches from trunk. Any steps we
should follow?
Another question from me: the new branch is created only for this JIRA but not
shared with other RBF feature work( e.g. RBF security), right?
> RBF: Add missing ClientProtocol APIs to RBF
> ---
>
> Key: HDFS-13655
> URL: https://issues.apache.org/jira/browse/HDFS-13655
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Xiao Chen
>Priority: Major
>
> As
> [discussed|https://issues.apache.org/jira/browse/HDFS-12858?focusedCommentId=16500975=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16500975|#comment-16500975]
> with [~elgoiri], there are some HDFS methods that does not take path as a
> parameter. We should support these to work with federation.
> The ones missing are:
> * Snapshots
> * Storage policies
> * Encryption zones
> * Cache pools
> One way to reasonably have them to work with federation is to 'list' each
> nameservice and concat the results. This can be done pretty much the same as
> {{refreshNodes()}} and it would be a matter of querying all the subclusters
> and aggregate the output (e.g., {{getDatanodeReport()}}.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13829) Needn't get min value of scan index and length for blockpoolReport
liaoyuxiangqin created HDFS-13829:
-
Summary: Needn't get min value of scan index and length for
blockpoolReport
Key: HDFS-13829
URL: https://issues.apache.org/jira/browse/HDFS-13829
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.2.0
Environment:
Reporter: liaoyuxiangqin
Assignee: liaoyuxiangqin
When i read the scan() of DirectoryScanner class in datanode, i found the
following condition code could be more simplify and easy to understand.
DirectoryScanner.java
{code:java}
if (d < blockpoolReport.length) {
// There may be multiple on-disk records for the same block, don't
increment
// the memory record pointer if so.
ScanInfo nextInfo = blockpoolReport[Math.min(d,
blockpoolReport.length - 1)];
if (nextInfo.getBlockId() != info.getBlockId()) {
++m;
}
} else {
++m;
}
{code}
as described above code segmet, i find the code of d < blockpoolReport.length
and the max of d is blockpoolReport.length-1, so that result of Math.min(d,
blockpoolReport.length - 1) always is d, so i think needn't to get min value of
scan index and length for blockpoolReport.
thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12862) CacheDirective may invalidata,when NN restart or make a transition to Active.
[
https://issues.apache.org/jira/browse/HDFS-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581859#comment-16581859
]
Wang XL commented on HDFS-12862:
check failed UT(TestNameNodeMetadataConsistency, TestCacheDirectives) and test
locally, it seems to work fine, ping [~daryn] , [~hexiaoqiao], [~jojochuang]
any further more suggestions?
> CacheDirective may invalidata,when NN restart or make a transition to Active.
> -
>
> Key: HDFS-12862
> URL: https://issues.apache.org/jira/browse/HDFS-12862
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching, hdfs
>Affects Versions: 2.7.1
> Environment:
>Reporter: Wang XL
>Priority: Major
> Labels: patch
> Attachments: HDFS-12862-branch-2.7.1.001.patch,
> HDFS-12862-trunk.002.patch, HDFS-12862-trunk.003.patch
>
>
> The logic in FSNDNCacheOp#modifyCacheDirective is not correct. when modify
> cacheDirective,the expiration in directive may be a relative expiryTime, and
> EditLog will serial a relative expiry time.
> {code:java}
> // Some comments here
> static void modifyCacheDirective(
> FSNamesystem fsn, CacheManager cacheManager, CacheDirectiveInfo
> directive,
> EnumSet flags, boolean logRetryCache) throws IOException {
> final FSPermissionChecker pc = getFsPermissionChecker(fsn);
> cacheManager.modifyDirective(directive, pc, flags);
> fsn.getEditLog().logModifyCacheDirectiveInfo(directive, logRetryCache);
> }
> {code}
> But when SBN replay the log ,it will invoke
> FSImageSerialization#readCacheDirectiveInfo as a absolute expiryTime.It will
> result in the inconsistency .
> {code:java}
> public static CacheDirectiveInfo readCacheDirectiveInfo(DataInput in)
> throws IOException {
> CacheDirectiveInfo.Builder builder =
> new CacheDirectiveInfo.Builder();
> builder.setId(readLong(in));
> int flags = in.readInt();
> if ((flags & 0x1) != 0) {
> builder.setPath(new Path(readString(in)));
> }
> if ((flags & 0x2) != 0) {
> builder.setReplication(readShort(in));
> }
> if ((flags & 0x4) != 0) {
> builder.setPool(readString(in));
> }
> if ((flags & 0x8) != 0) {
> builder.setExpiration(
> CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)));
> }
> if ((flags & ~0xF) != 0) {
> throw new IOException("unknown flags set in " +
> "ModifyCacheDirectiveInfoOp: " + flags);
> }
> return builder.build();
> }
> {code}
> In other words， fsn.getEditLog().logModifyCacheDirectiveInfo(directive,
> logRetryCache) may serial a relative expiry time,But
> builder.setExpiration(CacheDirectiveInfo.Expiration.newAbsolute(readLong(in)))
>read it as a absolute expiryTime.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-8277) Safemode enter fails when Standby NameNode is down
[
https://issues.apache.org/jira/browse/HDFS-8277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianfei Jiang reassigned HDFS-8277:
---
Assignee: Jianfei Jiang (was: Surendra Singh Lilhore)
> Safemode enter fails when Standby NameNode is down
> --
>
> Key: HDFS-8277
> URL: https://issues.apache.org/jira/browse/HDFS-8277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Affects Versions: 2.6.0
> Environment: HDP 2.2.0
>Reporter: Hari Sekhon
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-8277-safemode-edits.patch, HDFS-8277.patch,
> HDFS-8277_1.patch, HDFS-8277_2.patch, HDFS-8277_3.patch, HDFS-8277_4.patch,
> HDFS-8277_5.patch
>
>
> HDFS fails to enter safemode when the Standby NameNode is down (eg. due to
> AMBARI-10536).
> {code}hdfs dfsadmin -safemode enter
> safemode: Call From nn2/x.x.x.x to nn1:8020 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused{code}
> This appears to be a bug in that it's not trying both NameNodes like the
> standard hdfs client code does, and is instead stopping after getting a
> connection refused from nn1 which is down. I verified normal hadoop fs writes
> and reads via cli did work at this time, using nn2. I happened to run this
> command as the hdfs user on nn2 which was the surviving Active NameNode.
> After I re-bootstrapped the Standby NN to fix it the command worked as
> expected again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8277) Safemode enter fails when Standby NameNode is down
[
https://issues.apache.org/jira/browse/HDFS-8277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jianfei Jiang updated HDFS-8277:
Status: Patch Available (was: Open)
> Safemode enter fails when Standby NameNode is down
> --
>
> Key: HDFS-8277
> URL: https://issues.apache.org/jira/browse/HDFS-8277
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Affects Versions: 2.6.0
> Environment: HDP 2.2.0
>Reporter: Hari Sekhon
>Assignee: Jianfei Jiang
>Priority: Major
> Attachments: HDFS-8277-safemode-edits.patch, HDFS-8277.patch,
> HDFS-8277_1.patch, HDFS-8277_2.patch, HDFS-8277_3.patch, HDFS-8277_4.patch,
> HDFS-8277_5.patch
>
>
> HDFS fails to enter safemode when the Standby NameNode is down (eg. due to
> AMBARI-10536).
> {code}hdfs dfsadmin -safemode enter
> safemode: Call From nn2/x.x.x.x to nn1:8020 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused{code}
> This appears to be a bug in that it's not trying both NameNodes like the
> standard hdfs client code does, and is instead stopping after getting a
> connection refused from nn1 which is down. I verified normal hadoop fs writes
> and reads via cli did work at this time, using nn2. I happened to run this
> command as the hdfs user on nn2 which was the surviving Active NameNode.
> After I re-bootstrapped the Standby NN to fix it the command worked as
> expected again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-351) Add chill mode state to SCM
Ajay Kumar created HDDS-351: --- Summary: Add chill mode state to SCM Key: HDDS-351 URL: https://issues.apache.org/jira/browse/HDDS-351 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Ajay Kumar Add chill mode state to SCM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-351) Add chill mode state to SCM
[ https://issues.apache.org/jira/browse/HDDS-351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar reassigned HDDS-351: --- Assignee: Ajay Kumar > Add chill mode state to SCM > --- > > Key: HDDS-351 > URL: https://issues.apache.org/jira/browse/HDDS-351 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > Add chill mode state to SCM -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-350) ContainerMapping#flushContainerInfo doesn't set containerId
[ https://issues.apache.org/jira/browse/HDDS-350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar reassigned HDDS-350: --- Assignee: Ajay Kumar > ContainerMapping#flushContainerInfo doesn't set containerId > --- > > Key: HDDS-350 > URL: https://issues.apache.org/jira/browse/HDDS-350 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > > ContainerMapping#flushContainerInfo doesn't set containerId which results in > containerId being null in flushed containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-350) ContainerMapping#flushContainerInfo doesn't set containerId
Ajay Kumar created HDDS-350: --- Summary: ContainerMapping#flushContainerInfo doesn't set containerId Key: HDDS-350 URL: https://issues.apache.org/jira/browse/HDDS-350 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Ajay Kumar ContainerMapping#flushContainerInfo doesn't set containerId which results in containerId being null in flushed containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[
https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581819#comment-16581819
]
genericqa commented on HDFS-13746:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 19 unchanged - 5 fixed = 19 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-13746 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935763/HDFS-13746.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 77895654e77c 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7dc79a8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24785/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24785/testReport/ |
| Max. process+thread count | 3362 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console
[jira] [Commented] (HDFS-13822) speedup libhdfs++ build (enable parallel build)
[ https://issues.apache.org/jira/browse/HDFS-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581735#comment-16581735 ] Allen Wittenauer commented on HDFS-13822: - So gained 30 minutes just in the patch phase. Probably expect close to another 20 or so minutes after commit. > speedup libhdfs++ build (enable parallel build) > --- > > Key: HDFS-13822 > URL: https://issues.apache.org/jira/browse/HDFS-13822 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Pradeep Ambati >Priority: Minor > Attachments: HDFS-13382.000.patch, HDFS-13822.01.patch, > HDFS-13822.02.patch > > > libhdfs++ has significantly increased clean build times for the native client > on trunk. Problem is that libhdfs++ isn't build in parallel. When I tried to > force a parallel build by specifying -Dnative_make_args=-j4, the build fails > due to dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11481) hdfs snapshotDiff /.reserved/raw/... fails on snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-11481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581734#comment-16581734
]
genericqa commented on HDFS-11481:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.6.0 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
6s{color} | {color:red} root in branch-2.6.0 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
55s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v1.8.0_181. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v9-internal. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} branch-2.6.0 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v1.8.0_181. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v9-internal. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
11s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_181.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 11s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_181. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 10s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_181.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v9-internal.
{color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
Image:yetus/hadoop:date2018-08-15 |
| JIRA Issue | HDFS-11481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856755/HDFS-11481-branch-2.6.0.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8c47ea6c747e 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.6.0 / e349649 |
|
[jira] [Commented] (HDFS-13779) Implement performFailover logic for ObserverReadProxyProvider.
[
https://issues.apache.org/jira/browse/HDFS-13779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581729#comment-16581729
]
Chen Liang commented on HDFS-13779:
---
I see, so I guess the incremented {{currentIndex}} only takes effect when
{{invoke}} gets called next time, by which time {{getFilteredProxies}} will
return a different list with those proxies being removed. This is the part I
was missing, thanks for the clarification!
But this led me think about another situation: it seems to me that {{getProxy}}
can be called at any point of time. And {{getProxy}} will trigger
{{getAllProxies}} which further triggers {{refreshProxyState}}. And
{{refreshProxyState}} calls {{refreshState}} of each NN, which may change
{{isObserver}} to either true or false. In short, the value of {{isObserver}}
could change at any point of time due to {{getProxy}}. So when {{invoke}} gets
called each time, there seems no guarantee what will (not) be in the list
returned from {{getFilteredProxies}} with regard to the previous state. Is this
a valid case?
> Implement performFailover logic for ObserverReadProxyProvider.
> --
>
> Key: HDFS-13779
> URL: https://issues.apache.org/jira/browse/HDFS-13779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-13779-HDFS-12943.WIP00.patch
>
>
> Currently {{ObserverReadProxyProvider}} inherits {{performFailover()}} method
> from {{ConfiguredFailoverProxyProvider}}, which simply increments the index
> and switches over to another NameNode. The logic for ORPP should be smart
> enough to choose another observer, otherwise it can switch to a SBN, where
> reads are disallowed, or to an ANN, which defeats the purpose of reads from
> standby.
> This was discussed in HDFS-12976.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13822) speedup libhdfs++ build (enable parallel build)
[
https://issues.apache.org/jira/browse/HDFS-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581726#comment-16581726
]
genericqa commented on HDFS-13822:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 6m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 32m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
91m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 16m
9s{color} | {color:green} root generated 0 new + 9 unchanged - 2 fixed = 9
total (was 11) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
30s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 25s{color}
| {color:red} hadoop-hdfs-native-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed CTEST tests | test_test_libhdfs_threaded_hdfs_static |
| | test_libhdfs_threaded_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-13822 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935751/HDFS-13822.02.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit javadoc
mvninstall shadedclient xml |
| uname | Linux 23131ec12456 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d951af2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| CTEST |
https://builds.apache.org/job/PreCommit-HDFS-Build/24784/artifact/out/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-ctest.txt
|
[jira] [Commented] (HDFS-11481) hdfs snapshotDiff /.reserved/raw/... fails on snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-11481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581722#comment-16581722
]
Arpit Agarwal commented on HDFS-11481:
--
Should this be resolved?
> hdfs snapshotDiff /.reserved/raw/... fails on snapshottable directories
> ---
>
> Key: HDFS-11481
> URL: https://issues.apache.org/jira/browse/HDFS-11481
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
>Priority: Minor
> Attachments: HDFS-11481-branch-2.6.0.001.patch, HDFS-11481.001.patch,
> HDFS-11481.002.patch
>
>
> Successful command:
> {code}
> #> hdfs snapshotDiff /tmp/dir s1 s2
> Difference between snapshot s1 and snapshot s2 under directory /tmp/dir:
> M .
> + ./file1.txt
> {code}
> Unsuccessful command:
> {code}
> #> hdfs snapshotDiff /.reserved/raw/tmp/dir s1 s2
> snapshotDiff: Directory does not exist: /.reserved/raw/tmp/dir
> {code}
> Prefixing with raw path should run successfully and return same output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581698#comment-16581698 ] Daniel Templeton commented on HDFS-13746: - Thanks, [~smeng]. LGTM. +1 pending Jenkins. > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch, HDFS-13746.007.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-13746: -- Status: In Progress (was: Patch Available) > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch, HDFS-13746.007.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-13746: -- Release Note: Removed unused imports in rev 007. (was: Removed redundant imports in rev 007.) Attachment: HDFS-13746.007.patch Status: Patch Available (was: In Progress) > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch, HDFS-13746.007.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-13746: -- Attachment: (was: HDFS-13746.007.patch) > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581683#comment-16581683 ] Siyao Meng commented on HDFS-13746: --- [~templedf] Okay. Removed two unused imports in rev 007. Pending jenkins. > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-13746: -- Release Note: Removed redundant imports in rev 007. Attachment: HDFS-13746.007.patch Status: Patch Available (was: In Progress) > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch, HDFS-13746.007.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-13746: -- Status: In Progress (was: Patch Available) > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581673#comment-16581673 ] Daniel Templeton commented on HDFS-13746: - Last thing, checkstyle is reporting unused imports. > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581653#comment-16581653 ] Siyao Meng commented on HDFS-13746: --- +1 jenkins. unrelated tests. > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[
https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581651#comment-16581651
]
genericqa commented on HDFS-13746:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 19 unchanged - 5 fixed = 21 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.fs.viewfs.TestViewFileSystemHdfs |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-13746 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935739/HDFS-13746.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e80fd64a9200 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d951af2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/24783/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24783/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24783/testReport/ |
| Max.
[jira] [Commented] (HDFS-13818) Extend OIV to detect FSImage corruption
[ https://issues.apache.org/jira/browse/HDFS-13818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581619#comment-16581619 ] Arpit Agarwal commented on HDFS-13818: -- Thanks for filing this [~adam.antal]. It's a great idea to to detect the corruption earlier. The snapshot corruption we saw in HDFS-13314 is detected when the quota compute thread would walk the filesystem tree _after_ the image was successfully loaded. So I think the OIV will have to duplicate the same code paths NameNode executes at startup. Perhaps it will be easier to just try loading the image on another node. > Extend OIV to detect FSImage corruption > --- > > Key: HDFS-13818 > URL: https://issues.apache.org/jira/browse/HDFS-13818 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > > A follow-up Jira for HDFS-13031: an improvement of the OIV is suggested for > detecting corruptions like HDFS-13101 in an offline way. > The reasoning is the following. Apart from a NN startup throwing the error, > there is nothing in the customer's hand that could reassure him/her that the > FSImages is good or corrupted. > Although real full checking of the FSImage is only possible by the NN, for > stack traces associated with the observed corruption cases the solution of > putting up a tertiary NN is a little bit of overkill. The OIV would be a > handy choice, already having functionality like loading the fsimage and > constructing the folder structure, we just have to add the option of > detecting the null INodes. For e.g. the Delimited OIV processor can already > use in disk MetadataMap, which reduces memory consumption. Also there may be > a window for parallelizing: iterating through INodes for e.g. could be done > distributed, increasing efficiency, and we wouldn't need a high mem-high CPU > setup for just checking the FSImage. > The suggestion is to add a --detectCorruption option to the OIV which would > check the FSImage for consistency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13732) ECAdmin should print the policy name when an EC policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581597#comment-16581597 ] Hudson commented on HDFS-13732: --- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #14780 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14780/]) HDFS-13732. ECAdmin should print the policy name when an EC policy is (xiao: rev 7dc79a8b5b7af0bf37d25a221be8ed446b0edb74) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/ECAdmin.java > ECAdmin should print the policy name when an EC policy is set > - > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Fix For: 3.2.0 > > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13732) ECAdmin should print the policy name when an EC policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-13732: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) Committed to trunk > ECAdmin should print the policy name when an EC policy is set > - > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Fix For: 3.2.0 > > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13732) ECAdmin should print the policy name when an EC policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581594#comment-16581594 ] Xiao Chen commented on HDFS-13732: -- +1. While making an additional RPC is the only way to know the policy, I think it's helpful for an admin tool to print this information for supportability reasons. Thanks [~SoumyaPN] creating the issue, Zsolt for the fix and Kitti for the review! > ECAdmin should print the policy name when an EC policy is set > - > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13779) Implement performFailover logic for ObserverReadProxyProvider.
[
https://issues.apache.org/jira/browse/HDFS-13779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581593#comment-16581593
]
Erik Krogen edited comment on HDFS-13779 at 8/15/18 8:53 PM:
-
Thanks for looking [~vagarychen]!
For #1, the {{currentIndex}} only increases, then you take the modulo vs. the
current list size to get the actual index. So in the case you've discussed with
{{currentIndex = 2}} (not 3, since it is 0-indexed), before we have (caret
indicating current index):
{code:java}
1 2 3 4 5
^
{code}
Then we try 3 and 4, they both throw standby so we remove them, so the
remaining list is ({{currentIndex = 2}} still):
{code:java}
1 2 5
^
{code}
Next we try 5, is says RETRY, so we increment to get {{currentIndex = 3}}, then
we try 1 ({{3 % 3 == 0}}) as expected.
I went with {{currentIndex}} increasing and taking modulo, rather than always
maintaining an actual index, since the size of the list may change and
otherwise we would have to potentially update {{currentIndex}} every time the
list size changes (to avoid AIOOBE). Modulus gives us AIOOBE avoidance for free.
For #2, good point. For #3 / #4, yes, it needs cleanup, just a WIP for now.
was (Author: xkrogen):
Thanks for looking [~vagarychen]!
For #1, the {{currentIndex}} only increases, then you take the modulo vs. the
current list size to get the actual index. So in the case you've discussed with
{{currentIndex = 2}} (not 3, since it is 0-indexed), before we have (caret
indicating current index):
{code}
1 2 3 4 5
^
{code}
Then we try 3 and 4, they both throw standby so we remove them, so the
remaining list is ({{currentIndex = 2}} still):
{code}
1 2 5
^
{code}
Next we try 5, is says RETRY, so we increment to get {{currentIndex = 3}}, then
we try 1 ({{3 % 3 == 0}}) as expected.
I went with {{currentIndex}} increasing and taking modulo, rather than always
maintaining an actual index, since the size of the list may change and
otherwise we would have to potentially update {{currentIndex}} every time the
list size changes (to avoid AIOOBE). Modulus gives us AIOOBE avoidance for free.
For #2, good point. For #3 / #4, yes, it needs cleanup, just a WIP for now.
> Implement performFailover logic for ObserverReadProxyProvider.
> --
>
> Key: HDFS-13779
> URL: https://issues.apache.org/jira/browse/HDFS-13779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-13779-HDFS-12943.WIP00.patch
>
>
> Currently {{ObserverReadProxyProvider}} inherits {{performFailover()}} method
> from {{ConfiguredFailoverProxyProvider}}, which simply increments the index
> and switches over to another NameNode. The logic for ORPP should be smart
> enough to choose another observer, otherwise it can switch to a SBN, where
> reads are disallowed, or to an ANN, which defeats the purpose of reads from
> standby.
> This was discussed in HDFS-12976.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13779) Implement performFailover logic for ObserverReadProxyProvider.
[
https://issues.apache.org/jira/browse/HDFS-13779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581593#comment-16581593
]
Erik Krogen commented on HDFS-13779:
Thanks for looking [~vagarychen]!
For #1, the {{currentIndex}} only increases, then you take the modulo vs. the
current list size to get the actual index. So in the case you've discussed with
{{currentIndex = 2}} (not 3, since it is 0-indexed), before we have (caret
indicating current index):
{code}
1 2 3 4 5
^
{code}
Then we try 3 and 4, they both throw standby so we remove them, so the
remaining list is ({{currentIndex = 2}} still):
{code}
1 2 5
^
{code}
Next we try 5, is says RETRY, so we increment to get {{currentIndex = 3}}, then
we try 1 ({{3 % 3 == 0}}) as expected.
I went with {{currentIndex}} increasing and taking modulo, rather than always
maintaining an actual index, since the size of the list may change and
otherwise we would have to potentially update {{currentIndex}} every time the
list size changes (to avoid AIOOBE). Modulus gives us AIOOBE avoidance for free.
For #2, good point. For #3 / #4, yes, it needs cleanup, just a WIP for now.
> Implement performFailover logic for ObserverReadProxyProvider.
> --
>
> Key: HDFS-13779
> URL: https://issues.apache.org/jira/browse/HDFS-13779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-13779-HDFS-12943.WIP00.patch
>
>
> Currently {{ObserverReadProxyProvider}} inherits {{performFailover()}} method
> from {{ConfiguredFailoverProxyProvider}}, which simply increments the index
> and switches over to another NameNode. The logic for ORPP should be smart
> enough to choose another observer, otherwise it can switch to a SBN, where
> reads are disallowed, or to an ANN, which defeats the purpose of reads from
> standby.
> This was discussed in HDFS-12976.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13732) ECAdmin should print the policy name when an EC policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-13732: - Summary: ECAdmin should print the policy name when an EC policy is set (was: Erasure Coding policy name is not coming when the new policy is set) > ECAdmin should print the policy name when an EC policy is set > - > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13822) speedup libhdfs++ build (enable parallel build)
[ https://issues.apache.org/jira/browse/HDFS-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581590#comment-16581590 ] Allen Wittenauer commented on HDFS-13822: - -02: * cmake plugin for both builds * tie the clang build to a specific native-clange profile that can be configured in test-patch's personality for when it is appropriate * add dyld library paths for os x > speedup libhdfs++ build (enable parallel build) > --- > > Key: HDFS-13822 > URL: https://issues.apache.org/jira/browse/HDFS-13822 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Pradeep Ambati >Priority: Minor > Attachments: HDFS-13382.000.patch, HDFS-13822.01.patch, > HDFS-13822.02.patch > > > libhdfs++ has significantly increased clean build times for the native client > on trunk. Problem is that libhdfs++ isn't build in parallel. When I tried to > force a parallel build by specifying -Dnative_make_args=-j4, the build fails > due to dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13822) speedup libhdfs++ build (enable parallel build)
[ https://issues.apache.org/jira/browse/HDFS-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-13822: Attachment: HDFS-13822.02.patch > speedup libhdfs++ build (enable parallel build) > --- > > Key: HDFS-13822 > URL: https://issues.apache.org/jira/browse/HDFS-13822 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Pradeep Ambati >Priority: Minor > Attachments: HDFS-13382.000.patch, HDFS-13822.01.patch, > HDFS-13822.02.patch > > > libhdfs++ has significantly increased clean build times for the native client > on trunk. Problem is that libhdfs++ isn't build in parallel. When I tried to > force a parallel build by specifying -Dnative_make_args=-j4, the build fails > due to dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13671) Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
[
https://issues.apache.org/jira/browse/HDFS-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581525#comment-16581525
]
Kihwal Lee commented on HDFS-13671:
---
I've started working on the revert. So far in BlockManager, I had to deal with
the following:
{panel}
HDFS-11681 - retain the change in removeBlocksAssociatedTo()
HDFS-10301 - retain the change in BlockManager.
HDFS-10694 - retained logging changes. The change regarding sorted/unsorted was
removed in processReport() as it is no longer relevant.
HDFS-10858 - retain the change.
{panel}
I will post the revert patch and JIRAs involved in conflicts once everything is
done.
> Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet
> --
>
> Key: HDFS-13671
> URL: https://issues.apache.org/jira/browse/HDFS-13671
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.3
>Reporter: Yiqun Lin
>Priority: Major
>
> NameNode hung when deleting large files/blocks. The stack info:
> {code}
> "IPC Server handler 4 on 8020" #87 daemon prio=5 os_prio=0
> tid=0x7fb505b27800 nid=0x94c3 runnable [0x7fa861361000]
>java.lang.Thread.State: RUNNABLE
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.compare(FoldedTreeSet.java:474)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.removeAndGet(FoldedTreeSet.java:849)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet.remove(FoldedTreeSet.java:911)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.removeBlock(DatanodeStorageInfo.java:252)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:194)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.removeBlock(BlocksMap.java:108)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlockFromMap(BlockManager.java:3813)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.removeBlock(BlockManager.java:3617)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.removeBlocks(FSNamesystem.java:4270)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:4244)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInt(FSNamesystem.java:4180)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:4164)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:871)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.delete(AuthorizationProviderProxyClientProtocol.java:311)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:625)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
> {code}
> In the current deletion logic in NameNode, there are mainly two steps:
> * Collect INodes and all blocks to be deleted, then delete INodes.
> * Remove blocks chunk by chunk in a loop.
> Actually the first step should be a more expensive operation and will takes
> more time. However, now we always see NN hangs during the remove block
> operation.
> Looking into this, we introduced a new structure {{FoldedTreeSet}} to have a
> better performance in dealing FBR/IBRs. But compared with early
> implementation in remove-block logic, {{FoldedTreeSet}} seems more slower
> since It will take additional time to balance tree node. When there are large
> block to be removed/deleted, it looks bad.
> For the get type operations in {{DatanodeStorageInfo}}, we only provide the
> {{getBlockIterator}} to return blocks iterator and no other get operation
> with specified block. Still we need to use {{FoldedTreeSet}} in
> {{DatanodeStorageInfo}}? As we know {{FoldedTreeSet}} is benefit for Get not
> Update. Maybe we can revert this to the early implementation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13779) Implement performFailover logic for ObserverReadProxyProvider.
[
https://issues.apache.org/jira/browse/HDFS-13779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581498#comment-16581498
]
Chen Liang commented on HDFS-13779:
---
Thanks for the patch [~xkrogen], looking good to me overall, some comments:
1. I was too sure about the comment
{code}
// Only move the index if a new node should be tried.
// Don't move the index if the node will be removed from the list
// of observers, else one node will be skipped.
{code}
For example, it seems to me that, say there are 5 proxies and {{currentIndex}}
is currently 3. If no succeed, the for loop will traverse all 5 in the order of
3, 4, 5, 1, 2. Say 3, 4 both throw {{isStandbyException}} and removed from
list, then for 5 the policy returns RETRY. Then {{currentIndex}} gets
incremented by 1 to 4? Seems setting to 1 in this case makes more sense?
2. {{ObserverReadProxyProvider#getProxy}} the current code inserts a ','
between proxy strings, seems this is removed in the patch. Should we keep it?
3. need adding some new lines in class {{NameNodeProxyInfo}}
4. would be great if we can add a unit test.
> Implement performFailover logic for ObserverReadProxyProvider.
> --
>
> Key: HDFS-13779
> URL: https://issues.apache.org/jira/browse/HDFS-13779
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Konstantin Shvachko
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-13779-HDFS-12943.WIP00.patch
>
>
> Currently {{ObserverReadProxyProvider}} inherits {{performFailover()}} method
> from {{ConfiguredFailoverProxyProvider}}, which simply increments the index
> and switches over to another NameNode. The logic for ORPP should be smart
> enough to choose another observer, otherwise it can switch to a SBN, where
> reads are disallowed, or to an ANN, which defeats the purpose of reads from
> standby.
> This was discussed in HDFS-12976.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13825) HDFS Uses very outdated okhttp library
[
https://issues.apache.org/jira/browse/HDFS-13825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581491#comment-16581491
]
Sean Busbey commented on HDFS-13825:
[~BenParker25] is this problem still present if you rely on the
{{hadoop-client-api}} / {{hadoop-client-runtime}} jars? those are meant to be
the downstream facing interface for HDFS.
> HDFS Uses very outdated okhttp library
> --
>
> Key: HDFS-13825
> URL: https://issues.apache.org/jira/browse/HDFS-13825
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.3
>Reporter: Ben Parker
>Priority: Minor
>
> HDFS Client uses okHttp library version 2.7.4 which is two years out of date.
> [https://mvnrepository.com/artifact/com.squareup.okhttp/okhttp]
> The updates for this library have been moved to a new package here:
> [https://mvnrepository.com/artifact/com.squareup.okhttp3/okhttp]
>
> This causes dependancy management problems for services that use HDFS.
> For example trying to use okHttp in code that runs on Amazon EMR gives you
> Method not found errors due to the new version being kicked out in favour of
> the one used by HDFS.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-13746: -- Status: In Progress (was: Patch Available) > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581480#comment-16581480 ] Siyao Meng commented on HDFS-13746: --- [~templedf] You're right! I tested the first and second way. Both could still overflow without throwing exceptions. I have fixed it in rev 006 using the third way. Thanks! > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[ https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-13746: -- Attachment: HDFS-13746.006.patch Status: Patch Available (was: In Progress) > Still occasional "Should be different group" failure in > TestRefreshUserMappings#testGroupMappingRefresh > --- > > Key: HDFS-13746 > URL: https://issues.apache.org/jira/browse/HDFS-13746 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch, > HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch, > HDFS-13746.006.patch > > > In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of > time in sleep() helps but the problem still appears, which is annoying. > > Solution: > Use a loop to allow the test case to fail maxTrials times before declaring > failure. Wait 50 ms between each retry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-119) Skip Apache license header check for some ozone doc scripts
[
https://issues.apache.org/jira/browse/HDDS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDDS-119:
-
Environment: (was: {code}
Lines that start with ? in the ASF License report indicate files that do
not have an Apache license header: !?
/testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
/testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
!?
/testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
!?
/testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
!?
/testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
!?
/testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
!?
/testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
!?
/testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
!? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
!? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
{code})
> Skip Apache license header check for some ozone doc scripts
> ---
>
> Key: HDDS-119
> URL: https://issues.apache.org/jira/browse/HDDS-119
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: document
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-119.00.patch, HDDS-119.01.patch, HDDS-119.02.patch
>
>
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13822) speedup libhdfs++ build (enable parallel build)
[ https://issues.apache.org/jira/browse/HDFS-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581423#comment-16581423 ] Allen Wittenauer commented on HDFS-13822: - So I'm working on some Yetus bugs when I stumble back into this... mess. I just noticed the altern code. It triggers when the 'test-patch' profile is set, regardless of when native is set. This means it runs the clunky antrun version of cmake a few extra times during precommits that touch hdfs-native-client and any of its parents. Let's add this up. 1 x mvn install branch 2 x mvn compile branch 1 x mvn install patch 2 x mvn compile patch I'm flabbergasted. Especially so since it doesn't even work correctly. It's still using the same compilers for both compiles. > speedup libhdfs++ build (enable parallel build) > --- > > Key: HDFS-13822 > URL: https://issues.apache.org/jira/browse/HDFS-13822 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Pradeep Ambati >Priority: Minor > Attachments: HDFS-13382.000.patch, HDFS-13822.01.patch > > > libhdfs++ has significantly increased clean build times for the native client > on trunk. Problem is that libhdfs++ isn't build in parallel. When I tried to > force a parallel build by specifying -Dnative_make_args=-j4, the build fails > due to dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-119) Skip Apache license header check for some ozone doc scripts
[
https://issues.apache.org/jira/browse/HDDS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581404#comment-16581404
]
Hudson commented on HDDS-119:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14778 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14778/])
Revert "HDDS-119:Skip Apache license header check for some ozone doc
(aengineer: rev c918d88a9b5396a4ef8cf6f9aa323fb70c5c67f4)
* (edit) hadoop-build-tools/src/main/resources/checkstyle/suppressions.xml
* (edit) hadoop-ozone/pom.xml
> Skip Apache license header check for some ozone doc scripts
> ---
>
> Key: HDDS-119
> URL: https://issues.apache.org/jira/browse/HDDS-119
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: document
> Environment: {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-119.00.patch, HDDS-119.01.patch, HDDS-119.02.patch
>
>
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Reopened] (HDDS-119) Skip Apache license header check for some ozone doc scripts
[
https://issues.apache.org/jira/browse/HDDS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer reopened HDDS-119:
---
> Skip Apache license header check for some ozone doc scripts
> ---
>
> Key: HDDS-119
> URL: https://issues.apache.org/jira/browse/HDDS-119
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: document
> Environment: {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-119.00.patch, HDDS-119.01.patch, HDDS-119.02.patch
>
>
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-119) Skip Apache license header check for some ozone doc scripts
[
https://issues.apache.org/jira/browse/HDDS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581376#comment-16581376
]
Anu Engineer commented on HDDS-119:
---
I have just reverted this commit.
commit c918d88a9b5396a4ef8cf6f9aa323fb70c5c67f4 (HEAD -> trunk, origin/trunk,
origin/HEAD)
Author: Anu Engineer
Date: Wed Aug 15 10:06:17 2018 -0700
Revert "HDDS-119:Skip Apache license header check for some ozone doc
scripts. Contributed by Ajay Kumar"
This reverts commit c4bbcd5634d9e32183ae20adbeafca258568bedd.
> Skip Apache license header check for some ozone doc scripts
> ---
>
> Key: HDDS-119
> URL: https://issues.apache.org/jira/browse/HDDS-119
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: document
> Environment: {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-119.00.patch, HDDS-119.01.patch, HDDS-119.02.patch
>
>
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-119) Skip Apache license header check for some ozone doc scripts
[
https://issues.apache.org/jira/browse/HDDS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581359#comment-16581359
]
Anu Engineer edited comment on HDDS-119 at 8/15/18 5:15 PM:
Reverting this right now. Thanks for flagging it.
was (Author: anu):
Revert this right now. Thanks for flagging it.
> Skip Apache license header check for some ozone doc scripts
> ---
>
> Key: HDDS-119
> URL: https://issues.apache.org/jira/browse/HDDS-119
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: document
> Environment: {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-119.00.patch, HDDS-119.01.patch, HDDS-119.02.patch
>
>
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-119) Skip Apache license header check for some ozone doc scripts
[
https://issues.apache.org/jira/browse/HDDS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581359#comment-16581359
]
Anu Engineer commented on HDDS-119:
---
Revert this right now. Thanks for flagging it.
> Skip Apache license header check for some ozone doc scripts
> ---
>
> Key: HDDS-119
> URL: https://issues.apache.org/jira/browse/HDDS-119
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: document
> Environment: {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-119.00.patch, HDDS-119.01.patch, HDDS-119.02.patch
>
>
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13828) DataNode breaching Xceiver Count
[ https://issues.apache.org/jira/browse/HDFS-13828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581358#comment-16581358 ] Amithsha commented on HDFS-13828: - Moved from bug to task. > DataNode breaching Xceiver Count > > > Key: HDFS-13828 > URL: https://issues.apache.org/jira/browse/HDFS-13828 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode >Affects Versions: 2.7.1 >Reporter: Amithsha >Priority: Critical > > We were observing the breach of the xceiver count 4096, On a particular set > of nodes from 5 - 8 nodes in a 900 nodes cluster. > And we stopped the datanode services on those nodes and made to replicate > across the cluster. After that also, we observed the same issue on a new set > of nodes. > Q1: Why on a particular node, and also after decommissioning the node the > data should be replicated across the cluster, But why again difference set of > node? > Assumptions : > Reading a particular block/ data on that node might be the cause for this but > it should be mitigated after the decommission but not why? So suspected that > those MR jobs are triggered from Hive, so the query might be referring to the > same block mulitple times in different stages and creating this issue? > From Thread Dump : > Thread dump of datanode says that out of 4090+ xceiver threads created on > that node nearly 4000+ where belong to the same AppId of multiple mappers > with state no operation. > > Any suggestions on this? > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13828) DataNode breaching Xceiver Count
[ https://issues.apache.org/jira/browse/HDFS-13828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581356#comment-16581356 ] Amithsha commented on HDFS-13828: - Agree on xceiver count may be not sufficient but why for a particular node. And also its no on one node its on particular set of nodes. Adding the thread dump and datanode log. "DataXceiver for client DFSClient_attempt_1526704594842_1801529_m_008193_0_1144052212_1 at /x.x.x.x:38313 [Waiting for operation #28]" #55366018 daemon prio=5 os_prio=0 tid=0x7fcdaa0ca000 nid=0x128c runnable [0x7fcd24485000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000794fde658> (a sun.nio.ch.Util$2) - locked <0x000794fde640> (a java.util.Collections$UnmodifiableSet) - locked <0x0007b072db98> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x0005e9dc1de0> (a java.io.BufferedInputStream) at java.io.DataInputStream.readShort(DataInputStream.java:312) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) "DataXceiver for client DFSClient_attempt_1526704594842_1801529_m_003865_0_-1268040697_1 at /x.x.x.x:9258 [Sending block BP-1733841164-x.x.x.x-1440204182440:blk_8704233925_7644500095]" #55361352 daemon prio=5 os_prio=0 tid=0x7fcdaa36 nid=0xc93d runnable [0x7fcca559e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0007a5ae6e30> (a sun.nio.ch.Util$2) - locked <0x0007a5ae6e18> (a java.util.Collections$UnmodifiableSet) - locked <0x00079d242470> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > DataNode breaching Xceiver Count > > > Key: HDFS-13828 > URL: https://issues.apache.org/jira/browse/HDFS-13828 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Amithsha >Priority: Critical > > We were observing the breach of the xceiver count 4096, On a particular set > of nodes from 5 - 8 nodes in a 900 nodes cluster. > And we stopped the datanode services on those nodes and made to replicate > across the cluster. After that also, we observed the same issue on a new set > of nodes. > Q1: Why on a particular node, and also after decommissioning the node the > data should be replicated across the cluster, But why again difference set of > node? > Assumptions : > Reading a particular block/ data on that node might be the cause for this but > it should be mitigated after the decommission but not why? So suspected that > those MR jobs are triggered from Hive, so the query might be referring to the > same block mulitple times in different stages and creating this issue? > From Thread Dump : > Thread dump of datanode says that out of 4090+ xceiver threads created on > that node nearly 4000+ where belong to the same AppId of multiple mappers > with state no operation. > > Any suggestions on this? > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13828) DataNode breaching Xceiver Count
[ https://issues.apache.org/jira/browse/HDFS-13828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amithsha updated HDFS-13828: Issue Type: Task (was: Bug) > DataNode breaching Xceiver Count > > > Key: HDFS-13828 > URL: https://issues.apache.org/jira/browse/HDFS-13828 > Project: Hadoop HDFS > Issue Type: Task > Components: datanode >Affects Versions: 2.7.1 >Reporter: Amithsha >Priority: Critical > > We were observing the breach of the xceiver count 4096, On a particular set > of nodes from 5 - 8 nodes in a 900 nodes cluster. > And we stopped the datanode services on those nodes and made to replicate > across the cluster. After that also, we observed the same issue on a new set > of nodes. > Q1: Why on a particular node, and also after decommissioning the node the > data should be replicated across the cluster, But why again difference set of > node? > Assumptions : > Reading a particular block/ data on that node might be the cause for this but > it should be mitigated after the decommission but not why? So suspected that > those MR jobs are triggered from Hive, so the query might be referring to the > same block mulitple times in different stages and creating this issue? > From Thread Dump : > Thread dump of datanode says that out of 4090+ xceiver threads created on > that node nearly 4000+ where belong to the same AppId of multiple mappers > with state no operation. > > Any suggestions on this? > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13828) DataNode breaching Xceiver Count
[ https://issues.apache.org/jira/browse/HDFS-13828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581356#comment-16581356 ] Amithsha edited comment on HDFS-13828 at 8/15/18 5:06 PM: -- Agree on xceiver count may be not sufficient but why for a particular node. And also its not happening on one node it's on a particular set of nodes. Adding the thread dump and datanode log. "DataXceiver for client DFSClient_attempt_1526704594842_1801529_m_008193_0_1144052212_1 at /x.x.x.x:38313 [Waiting for operation #28|#28]" #55366018 daemon prio=5 os_prio=0 tid=0x7fcdaa0ca000 nid=0x128c runnable [0x7fcd24485000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000794fde658> (a sun.nio.ch.Util$2) - locked <0x000794fde640> (a java.util.Collections$UnmodifiableSet) - locked <0x0007b072db98> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x0005e9dc1de0> (a java.io.BufferedInputStream) at java.io.DataInputStream.readShort(DataInputStream.java:312) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) "DataXceiver for client DFSClient_attempt_1526704594842_1801529_m_003865_0_-1268040697_1 at /x.x.x.x:9258 [Sending block BP-1733841164-x.x.x.x-1440204182440:blk_8704233925_7644500095]" #55361352 daemon prio=5 os_prio=0 tid=0x7fcdaa36 nid=0xc93d runnable [0x7fcca559e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0007a5ae6e30> (a sun.nio.ch.Util$2) - locked <0x0007a5ae6e18> (a java.util.Collections$UnmodifiableSet) - locked <0x00079d242470> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) was (Author: amithsha): Agree on xceiver count may be not sufficient but why for a particular node. And also its no on one node its on particular set of nodes. Adding the thread dump and datanode log. "DataXceiver for client DFSClient_attempt_1526704594842_1801529_m_008193_0_1144052212_1 at /x.x.x.x:38313 [Waiting for operation #28]" #55366018 daemon prio=5 os_prio=0 tid=0x7fcdaa0ca000 nid=0x128c runnable [0x7fcd24485000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000794fde658> (a sun.nio.ch.Util$2) - locked <0x000794fde640> (a java.util.Collections$UnmodifiableSet) - locked <0x0007b072db98> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked <0x0005e9dc1de0> (a java.io.BufferedInputStream) at java.io.DataInputStream.readShort(DataInputStream.java:312) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at
[jira] [Commented] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[
https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581343#comment-16581343
]
Daniel Templeton commented on HDFS-13746:
-
Thanks, [~smeng], the patch looks great. One question:
{code}Math.toIntExact(groupRefreshTimeoutSec) * 1000 * 30{code}
Should that be
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000) * 30{code}
or maybe even
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000 * 30}
?
If you're going to proactively check for int overflows, it seems like you'd
want to do it for the final product rather than a factor. Thoughts?
> Still occasional "Should be different group" failure in
> TestRefreshUserMappings#testGroupMappingRefresh
> ---
>
> Key: HDFS-13746
> URL: https://issues.apache.org/jira/browse/HDFS-13746
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch,
> HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch
>
>
> In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of
> time in sleep() helps but the problem still appears, which is annoying.
>
> Solution:
> Use a loop to allow the test case to fail maxTrials times before declaring
> failure. Wait 50 ms between each retry.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[
https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581343#comment-16581343
]
Daniel Templeton edited comment on HDFS-13746 at 8/15/18 4:54 PM:
--
Thanks, [~smeng], the patch looks great. One question:
{code}Math.toIntExact(groupRefreshTimeoutSec) * 1000 * 30{code}
Should that be
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000) * 30{code}
or maybe even
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000 * 30{code}
?
If you're going to proactively check for int overflows, it seems like you'd
want to do it for the final product rather than a factor. Thoughts?
was (Author: templedf):
Thanks, [~smeng], the patch looks great. One question:
{code}Math.toIntExact(groupRefreshTimeoutSec) * 1000 * 30{code}
Should that be
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000) * 30{code}
or maybe even
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000 * 30}
?
If you're going to proactively check for int overflows, it seems like you'd
want to do it for the final product rather than a factor. Thoughts?
> Still occasional "Should be different group" failure in
> TestRefreshUserMappings#testGroupMappingRefresh
> ---
>
> Key: HDFS-13746
> URL: https://issues.apache.org/jira/browse/HDFS-13746
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch,
> HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch
>
>
> In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of
> time in sleep() helps but the problem still appears, which is annoying.
>
> Solution:
> Use a loop to allow the test case to fail maxTrials times before declaring
> failure. Wait 50 ms between each retry.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13746) Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
[
https://issues.apache.org/jira/browse/HDFS-13746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581343#comment-16581343
]
Daniel Templeton edited comment on HDFS-13746 at 8/15/18 4:54 PM:
--
Thanks, [~smeng], the patch looks great. One question:
{code}Math.toIntExact(groupRefreshTimeoutSec) * 1000 * 30{code}
Should that be
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000) * 30{code}
or maybe even
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000 * 30){code}
?
If you're going to proactively check for int overflows, it seems like you'd
want to do it for the final product rather than a factor. Thoughts?
was (Author: templedf):
Thanks, [~smeng], the patch looks great. One question:
{code}Math.toIntExact(groupRefreshTimeoutSec) * 1000 * 30{code}
Should that be
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000) * 30{code}
or maybe even
{code}Math.toIntExact(groupRefreshTimeoutSec * 1000 * 30{code}
?
If you're going to proactively check for int overflows, it seems like you'd
want to do it for the final product rather than a factor. Thoughts?
> Still occasional "Should be different group" failure in
> TestRefreshUserMappings#testGroupMappingRefresh
> ---
>
> Key: HDFS-13746
> URL: https://issues.apache.org/jira/browse/HDFS-13746
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Major
> Attachments: HDFS-13746.001.patch, HDFS-13746.002.patch,
> HDFS-13746.003.patch, HDFS-13746.004.patch, HDFS-13746.005.patch
>
>
> In https://issues.apache.org/jira/browse/HDFS-13723, increasing the amount of
> time in sleep() helps but the problem still appears, which is annoying.
>
> Solution:
> Use a loop to allow the test case to fail maxTrials times before declaring
> failure. Wait 50 ms between each retry.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-119) Skip Apache license header check for some ozone doc scripts
[
https://issues.apache.org/jira/browse/HDDS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581339#comment-16581339
]
Xiao Chen commented on HDDS-119:
As it turns out on HADOOP-15675, this commit has muted checkstyle on trunk
Can you please look into it and provide a fix or revert? Thanks a lot
> Skip Apache license header check for some ozone doc scripts
> ---
>
> Key: HDDS-119
> URL: https://issues.apache.org/jira/browse/HDDS-119
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: document
> Environment: {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-119.00.patch, HDDS-119.01.patch, HDDS-119.02.patch
>
>
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/theme.toml !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/fonts/glyphicons-halflings-regular.svg
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/bootstrap.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/js/jquery.min.js
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css.map
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap.min.css
> !?
> /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/static/css/bootstrap-theme.min.css.map
> !? /testptch/hadoop/hadoop-ozone/docs/themes/ozonedoc/layouts/index.html
> !? /testptch/hadoop/hadoop-ozone/docs/static/OzoneOverview.svg
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13217) Audit log all EC policy names during addErasureCodingPolicies
[
https://issues.apache.org/jira/browse/HDFS-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581336#comment-16581336
]
Hudson commented on HDFS-13217:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14777 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14777/])
HDFS-13217. Audit log all EC policy names during (xiao: rev
ef02f9b233116a10cf9274b3753fe0b8dcbe8d92)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
> Audit log all EC policy names during addErasureCodingPolicies
> -
>
> Key: HDFS-13217
> URL: https://issues.apache.org/jira/browse/HDFS-13217
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13217.001.patch, HDFS-13217.002.patch,
> HDFS-13217.003.patch, HDFS-13217.004.patch, HDFS-13217.005.patch
>
>
> When i read the addErasureCodingPolicies() of FSNamesystem class in namenode,
> i found the following code only used last ec policy name for logAuditEvent,
> i think this audit log can't track whole policies for the add multiple
> erasure coding policies to the ErasureCodingPolicyManager. Thanks.
> {code:java|title=FSNamesystem.java|borderStyle=solid}
> try {
> checkOperation(OperationCategory.WRITE);
> checkNameNodeSafeMode("Cannot add erasure coding policy");
> for (ErasureCodingPolicy policy : policies) {
> try {
> ErasureCodingPolicy newPolicy =
> FSDirErasureCodingOp.addErasureCodingPolicy(this, policy,
> logRetryCache);
> addECPolicyName = newPolicy.getName();
> responses.add(new AddErasureCodingPolicyResponse(newPolicy));
> } catch (HadoopIllegalArgumentException e) {
> responses.add(new AddErasureCodingPolicyResponse(policy, e));
> }
> }
> success = true;
> return responses.toArray(new AddErasureCodingPolicyResponse[0]);
> } finally {
> writeUnlock(operationName);
> if (success) {
> getEditLog().logSync();
> }
> logAuditEvent(success, operationName,addECPolicyName, null, null);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13217) Audit log all EC policy names during addErasureCodingPolicies
[
https://issues.apache.org/jira/browse/HDFS-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-13217:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
+1, failed test is unrelated.
Thanks [~liaoyuxiangqin] for reporting and fixing the issue. Also thanks
[~eddyxu] and [~knanasi] for the reviews!
> Audit log all EC policy names during addErasureCodingPolicies
> -
>
> Key: HDFS-13217
> URL: https://issues.apache.org/jira/browse/HDFS-13217
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HDFS-13217.001.patch, HDFS-13217.002.patch,
> HDFS-13217.003.patch, HDFS-13217.004.patch, HDFS-13217.005.patch
>
>
> When i read the addErasureCodingPolicies() of FSNamesystem class in namenode,
> i found the following code only used last ec policy name for logAuditEvent,
> i think this audit log can't track whole policies for the add multiple
> erasure coding policies to the ErasureCodingPolicyManager. Thanks.
> {code:java|title=FSNamesystem.java|borderStyle=solid}
> try {
> checkOperation(OperationCategory.WRITE);
> checkNameNodeSafeMode("Cannot add erasure coding policy");
> for (ErasureCodingPolicy policy : policies) {
> try {
> ErasureCodingPolicy newPolicy =
> FSDirErasureCodingOp.addErasureCodingPolicy(this, policy,
> logRetryCache);
> addECPolicyName = newPolicy.getName();
> responses.add(new AddErasureCodingPolicyResponse(newPolicy));
> } catch (HadoopIllegalArgumentException e) {
> responses.add(new AddErasureCodingPolicyResponse(policy, e));
> }
> }
> success = true;
> return responses.toArray(new AddErasureCodingPolicyResponse[0]);
> } finally {
> writeUnlock(operationName);
> if (success) {
> getEditLog().logSync();
> }
> logAuditEvent(success, operationName,addECPolicyName, null, null);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13217) Audit log all EC policy names during addErasureCodingPolicies
[
https://issues.apache.org/jira/browse/HDFS-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-13217:
-
Summary: Audit log all EC policy names during addErasureCodingPolicies
(was: Log audit event only used last EC policy name when add multiple policies
from file )
> Audit log all EC policy names during addErasureCodingPolicies
> -
>
> Key: HDFS-13217
> URL: https://issues.apache.org/jira/browse/HDFS-13217
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13217.001.patch, HDFS-13217.002.patch,
> HDFS-13217.003.patch, HDFS-13217.004.patch, HDFS-13217.005.patch
>
>
> When i read the addErasureCodingPolicies() of FSNamesystem class in namenode,
> i found the following code only used last ec policy name for logAuditEvent,
> i think this audit log can't track whole policies for the add multiple
> erasure coding policies to the ErasureCodingPolicyManager. Thanks.
> {code:java|title=FSNamesystem.java|borderStyle=solid}
> try {
> checkOperation(OperationCategory.WRITE);
> checkNameNodeSafeMode("Cannot add erasure coding policy");
> for (ErasureCodingPolicy policy : policies) {
> try {
> ErasureCodingPolicy newPolicy =
> FSDirErasureCodingOp.addErasureCodingPolicy(this, policy,
> logRetryCache);
> addECPolicyName = newPolicy.getName();
> responses.add(new AddErasureCodingPolicyResponse(newPolicy));
> } catch (HadoopIllegalArgumentException e) {
> responses.add(new AddErasureCodingPolicyResponse(policy, e));
> }
> }
> success = true;
> return responses.toArray(new AddErasureCodingPolicyResponse[0]);
> } finally {
> writeUnlock(operationName);
> if (success) {
> getEditLog().logSync();
> }
> logAuditEvent(success, operationName,addECPolicyName, null, null);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13828) DataNode breaching Xceiver Count
[ https://issues.apache.org/jira/browse/HDFS-13828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581298#comment-16581298 ] Wei-Chiu Chuang commented on HDFS-13828: It doesn't seem like a bug to me. There are scenarios where the default 4096 transxcever count may be exceeded, and depending on the scenario, you might want to raise the max tranxcever count. Also note that jira is not a forum for questions but for bug reports. If you think there are bugs, please attach relevant logs and setups. > DataNode breaching Xceiver Count > > > Key: HDFS-13828 > URL: https://issues.apache.org/jira/browse/HDFS-13828 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Amithsha >Priority: Critical > > We were observing the breach of the xceiver count 4096, On a particular set > of nodes from 5 - 8 nodes in a 900 nodes cluster. > And we stopped the datanode services on those nodes and made to replicate > across the cluster. After that also, we observed the same issue on a new set > of nodes. > Q1: Why on a particular node, and also after decommissioning the node the > data should be replicated across the cluster, But why again difference set of > node? > Assumptions : > Reading a particular block/ data on that node might be the cause for this but > it should be mitigated after the decommission but not why? So suspected that > those MR jobs are triggered from Hive, so the query might be referring to the > same block mulitple times in different stages and creating this issue? > From Thread Dump : > Thread dump of datanode says that out of 4090+ xceiver threads created on > that node nearly 4000+ where belong to the same AppId of multiple mappers > with state no operation. > > Any suggestions on this? > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-328) Support export and import of the KeyValueContainer
[ https://issues.apache.org/jira/browse/HDDS-328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581281#comment-16581281 ] Xiaoyu Yao commented on HDDS-328: - Thanks [~elek] for working on this. The patch v3 looks excellent to me overall. I just have few comments in addition to [~hanishakoneru]'s comment above. KeyValueContainer.java Line 387-388: these should be removed as it is a dup of line 383-385. Line 398: agree with Hanisha that we need to handle the error case. I'm OK if you plan to add that in HDDS-75. TarContainerPacker.java Have we consider adding other compressor support such as bzip or lz4 which has native codec optimization in existing hadoop code? Line 81/91: we should avoid using hard-coded path separators the applies only to *nix OS. Line 123: should we check if the tarInput.read return -1 (EOF) to avoid adding remaining negatively? Line 164-165: can we use try-with-resource to avoid leak in the exception case? > Support export and import of the KeyValueContainer > -- > > Key: HDDS-328 > URL: https://issues.apache.org/jira/browse/HDDS-328 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Critical > Fix For: 0.2.1 > > Attachments: HDDS-328.002.patch, HDDS-328.003.patch > > > In HDDS-75 we pack the container data to an archive file, copy to other > datanodes and create the container from the archive. > As I wrote in the comment of HDDS-75 I propose to separate the patch to make > it easier to review. > In this patch we need to extend the existing Container interface with adding > export/import methods to save the container data to one binary input/output > stream. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13819) TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky
[
https://issues.apache.org/jira/browse/HDFS-13819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated HDFS-13819:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
Status: Resolved (was: Patch Available)
Thanks for the review, [~elgoiri]! Committed to trunk.
> TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky
>
>
> Key: HDFS-13819
> URL: https://issues.apache.org/jira/browse/HDFS-13819
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-13819.001.patch, HDFS-13819.002.patch
>
>
> We're seeing the test fail periodically with:
> {quote}java.lang.AssertionError: expected:<2> but was:<1>{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-347) TestCloseContainerByPipeline#testCloseContainerViaStandaAlone sometimes fails
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581249#comment-16581249
]
Hudson commented on HDDS-347:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14775 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14775/])
HDDS-347. TestCloseContainerByPipeline#testCloseContainerViaStandaAlone (xyao:
rev f2315f2e9c31879b8c30eb9028f1959dfe48e5f4)
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/container/common/statemachine/commandhandler/TestCloseContainerByPipeline.java
> TestCloseContainerByPipeline#testCloseContainerViaStandaAlone sometimes fails
> -
>
> Key: HDDS-347
> URL: https://issues.apache.org/jira/browse/HDDS-347
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-347.000.patch, HDDS-347.001.patch
>
>
> This issue was finded in the automatic JenKins unit test of HDDS-265.
> The container life cycle state is : Open -> Closing -> closed, this test
> submit the container close command and wait for container state change to
> *not equal to open*, actually even when the state condition(not equal to
> open) is satisfied, the container may still in process of closing, so the LOG
> which will printf after the container closed can't be find sometimes and the
> test fails.
> {code:java|title=KeyValueContainer.java|borderStyle=solid}
> try {
> writeLock();
> containerData.closeContainer();
> File containerFile = getContainerFile();
> // update the new container data to .container File
> updateContainerFile(containerFile);
> } catch (StorageContainerException ex) {
> {code}
> Looking at the code above, the container state changes from CLOSING to CLOSED
> in the first step, the remaining *updateContainerFile* may take hundreds of
> milliseconds, so even we modify the test logic to wait for the *CLOSED* state
> will not guarantee the test success, too.
> These are two way to fix this:
> 1, Remove one of the double check which depends on the LOG.
> 2, If we have to preserve the double check, we should wait for the *CLOSED*
> state and sleep for a while to wait for the LOG appears.
> patch 000 is based on the second way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-347) TestCloseContainerByPipeline#testCloseContainerViaStandaAlone sometimes fails
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDDS-347:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks [~GeLiXin] for the contribution and all for the reviews. I've committed
the patch to trunk.
> TestCloseContainerByPipeline#testCloseContainerViaStandaAlone sometimes fails
> -
>
> Key: HDDS-347
> URL: https://issues.apache.org/jira/browse/HDDS-347
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-347.000.patch, HDDS-347.001.patch
>
>
> This issue was finded in the automatic JenKins unit test of HDDS-265.
> The container life cycle state is : Open -> Closing -> closed, this test
> submit the container close command and wait for container state change to
> *not equal to open*, actually even when the state condition(not equal to
> open) is satisfied, the container may still in process of closing, so the LOG
> which will printf after the container closed can't be find sometimes and the
> test fails.
> {code:java|title=KeyValueContainer.java|borderStyle=solid}
> try {
> writeLock();
> containerData.closeContainer();
> File containerFile = getContainerFile();
> // update the new container data to .container File
> updateContainerFile(containerFile);
> } catch (StorageContainerException ex) {
> {code}
> Looking at the code above, the container state changes from CLOSING to CLOSED
> in the first step, the remaining *updateContainerFile* may take hundreds of
> milliseconds, so even we modify the test logic to wait for the *CLOSED* state
> will not guarantee the test success, too.
> These are two way to fix this:
> 1, Remove one of the double check which depends on the LOG.
> 2, If we have to preserve the double check, we should wait for the *CLOSED*
> state and sleep for a while to wait for the LOG appears.
> patch 000 is based on the second way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-347) TestCloseContainerByPipeline#testCloseContainerViaStandaAlone sometimes fails
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDDS-347:
Summary: TestCloseContainerByPipeline#testCloseContainerViaStandaAlone
sometimes fails (was: Fix : testCloseContainerViaStandaAlone fails sometimes)
> TestCloseContainerByPipeline#testCloseContainerViaStandaAlone sometimes fails
> -
>
> Key: HDDS-347
> URL: https://issues.apache.org/jira/browse/HDDS-347
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-347.000.patch, HDDS-347.001.patch
>
>
> This issue was finded in the automatic JenKins unit test of HDDS-265.
> The container life cycle state is : Open -> Closing -> closed, this test
> submit the container close command and wait for container state change to
> *not equal to open*, actually even when the state condition(not equal to
> open) is satisfied, the container may still in process of closing, so the LOG
> which will printf after the container closed can't be find sometimes and the
> test fails.
> {code:java|title=KeyValueContainer.java|borderStyle=solid}
> try {
> writeLock();
> containerData.closeContainer();
> File containerFile = getContainerFile();
> // update the new container data to .container File
> updateContainerFile(containerFile);
> } catch (StorageContainerException ex) {
> {code}
> Looking at the code above, the container state changes from CLOSING to CLOSED
> in the first step, the remaining *updateContainerFile* may take hundreds of
> milliseconds, so even we modify the test logic to wait for the *CLOSED* state
> will not guarantee the test success, too.
> These are two way to fix this:
> 1, Remove one of the double check which depends on the LOG.
> 2, If we have to preserve the double check, we should wait for the *CLOSED*
> state and sleep for a while to wait for the LOG appears.
> patch 000 is based on the second way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-347) Fix : testCloseContainerViaStandaAlone fails sometimes
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581206#comment-16581206
]
Xiaoyu Yao commented on HDDS-347:
-
Thanks [~GeLiXin] for the update. +1 for v1 patch, I will commit it shortly.
> Fix : testCloseContainerViaStandaAlone fails sometimes
> --
>
> Key: HDDS-347
> URL: https://issues.apache.org/jira/browse/HDDS-347
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-347.000.patch, HDDS-347.001.patch
>
>
> This issue was finded in the automatic JenKins unit test of HDDS-265.
> The container life cycle state is : Open -> Closing -> closed, this test
> submit the container close command and wait for container state change to
> *not equal to open*, actually even when the state condition(not equal to
> open) is satisfied, the container may still in process of closing, so the LOG
> which will printf after the container closed can't be find sometimes and the
> test fails.
> {code:java|title=KeyValueContainer.java|borderStyle=solid}
> try {
> writeLock();
> containerData.closeContainer();
> File containerFile = getContainerFile();
> // update the new container data to .container File
> updateContainerFile(containerFile);
> } catch (StorageContainerException ex) {
> {code}
> Looking at the code above, the container state changes from CLOSING to CLOSED
> in the first step, the remaining *updateContainerFile* may take hundreds of
> milliseconds, so even we modify the test logic to wait for the *CLOSED* state
> will not guarantee the test success, too.
> These are two way to fix this:
> 1, Remove one of the double check which depends on the LOG.
> 2, If we have to preserve the double check, we should wait for the *CLOSED*
> state and sleep for a while to wait for the LOG appears.
> patch 000 is based on the second way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581104#comment-16581104
]
genericqa commented on HDDS-343:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
8s{color} | {color:red} hadoop-hdds/common generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} server-scm in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdds/common |
| |
org.apache.hadoop.ozone.common.statemachine.StateMachine.getTransition(Enum,
Enum) makes inefficient use of keySet iterator instead of entrySet iterator At
StateMachine.java:of keySet iterator instead of entrySet iterator At
StateMachine.java:[line 75] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-343 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935687/HDDS-343.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 753fab668d53 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581107#comment-16581107
]
genericqa commented on HDDS-343:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
6s{color} | {color:red} hadoop-hdds/common generated 1 new + 0 unchanged - 0
fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} server-scm in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdds/common |
| |
org.apache.hadoop.ozone.common.statemachine.StateMachine.getTransition(Enum,
Enum) makes inefficient use of keySet iterator instead of entrySet iterator At
StateMachine.java:of keySet iterator instead of entrySet iterator At
StateMachine.java:[line 75] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-343 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935687/HDDS-343.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 96de0705dc86 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Created] (HDFS-13828) DataNode breaching Xceiver Count
Amithsha created HDFS-13828: --- Summary: DataNode breaching Xceiver Count Key: HDFS-13828 URL: https://issues.apache.org/jira/browse/HDFS-13828 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.7.1 Reporter: Amithsha We were observing the breach of the xceiver count 4096, On a particular set of nodes from 5 - 8 nodes in a 900 nodes cluster. And we stopped the datanode services on those nodes and made to replicate across the cluster. After that also, we observed the same issue on a new set of nodes. Q1: Why on a particular node, and also after decommissioning the node the data should be replicated across the cluster, But why again difference set of node? Assumptions : Reading a particular block/ data on that node might be the cause for this but it should be mitigated after the decommission but not why? So suspected that those MR jobs are triggered from Hive, so the query might be referring to the same block mulitple times in different stages and creating this issue? >From Thread Dump : Thread dump of datanode says that out of 4090+ xceiver threads created on that node nearly 4000+ where belong to the same AppId of multiple mappers with state no operation. Any suggestions on this? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13732) Erasure Coding policy name is not coming when the new policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581092#comment-16581092 ] Kitti Nanasi commented on HDFS-13732: - Thanks [~SoumyaPN] for reporting the issue and [~zvenczel] for providing the patch! +1 (non-binding) The patch looks good to me. The test failures do seem to be unrelated. > Erasure Coding policy name is not coming when the new policy is set > --- > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13732) Erasure Coding policy name is not coming when the new policy is set
[
https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581061#comment-16581061
]
Zsolt Venczel commented on HDFS-13732:
--
Failed tests are passing locally with or without the patch therefore they
should be unrelated:
{code}
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.hdfs.client.impl.TestBlockReaderLocal
[INFO] Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.179
s - in org.apache.hadoop.hdfs.client.impl.TestBlockReaderLocal
[INFO] Running org.apache.hadoop.hdfs.TestSafeModeWithStripedFile
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.843 s
- in org.apache.hadoop.hdfs.TestSafeModeWithStripedFile
[INFO] Running org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.357 s
- in org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus
[INFO] Running
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 48.797 s
- in org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
[INFO] Running org.apache.hadoop.hdfs.TestMaintenanceState
[INFO] Tests run: 25, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 350.793
s - in org.apache.hadoop.hdfs.TestMaintenanceState
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 75, Failures: 0, Errors: 0, Skipped: 0
{code}
> Erasure Coding policy name is not coming when the new policy is set
> ---
>
> Key: HDFS-13732
> URL: https://issues.apache.org/jira/browse/HDFS-13732
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding, tools
>Affects Versions: 3.0.0
>Reporter: Soumyapn
>Assignee: Zsolt Venczel
>Priority: Trivial
> Attachments: EC_Policy.PNG, HDFS-13732.01.patch
>
>
> Scenerio:
> If the new policy apart from the default EC policy is set for the HDFS
> directory, then the console message is coming as "Set default erasure coding
> policy on "
> Expected output:
> It would be good If the EC policy name is displayed when the policy is set...
>
> Actual output:
> Set default erasure coding policy on
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581027#comment-16581027
]
Elek, Marton commented on HDDS-343:
---
I tested it with the byteman script (see HDDS-342 and HADOOP-15656) and found
that with this patch the containers are closed.
Used configuration:
{code}
OZONE-SITE.XML_ozone.scm.container.provision_batch_size=1
OZONE-SITE.XML_ozone.scm.container.size.gb=1
{code}
Used commands:
{code}
ozone oz -createVolume /vol1 -user hadoop --quota 1TB --root
ozone oz -createBucket /vol1/bucket
dd if=/dev/zero of=/tmp/test bs=1024000 count=512
ozone oz -putKey /vol1/bucket/file1 -file /tmp/test
ozone oz -infoKey /vol1/bucket/file1
ozone oz -putKey /vol1/bucket/file2 -file /tmp/test
{code}
ps: v2 is the same as v1 but I improved the error handling in case of the
transition is missing between two state.
> Containers are stuck in closing state in scm
>
>
> Key: HDDS-343
> URL: https://issues.apache.org/jira/browse/HDDS-343
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Fix For: 0.2.1
>
> Attachments: HDDS-343.001.patch, HDDS-343.002.patch
>
>
> Containers could not been closed currently.
> The datanode is closing the containers and sending the CLOSED state in the
> container report but SCM doesn't register that the state is closed and
> sending the close command again and again.
> I think the ContainerMapping.processContainerReport should be improved.
> {code}
> scm_1 | --> RPC message request: SCMHeartbeatRequestProto from
> 172.25.0.2:33912
> scm_1 | datanodeDetails {
> scm_1 | uuid: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 | ipAddress: "172.25.0.2"
> scm_1 | hostName: "365fd1f44f0b"
> scm_1 | ports {
> scm_1 | name: "STANDALONE"
> scm_1 | value: 9859
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "RATIS"
> scm_1 | value: 9858
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "REST"
> scm_1 | value: 9880
> scm_1 | }
> scm_1 | }
> scm_1 | nodeReport {
> scm_1 | storageReport {
> scm_1 | storageUuid: "DS-61e76107-85c5-437a-95a7-aeb8b3e7827f"
> scm_1 | storageLocation: "/tmp/hadoop-hadoop/dfs/data"
> scm_1 | capacity: 491630870528
> scm_1 | scmUsed: 2708828160
> scm_1 | remaining: 24263614464
> scm_1 | storageType: DISK
> scm_1 | failed: false
> scm_1 | }
> scm_1 | }
> scm_1 | containerReport {
> scm_1 | reports {
> scm_1 | containerID: 1
> scm_1 | used: 1061158912
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1061158912
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 2
> scm_1 | used: 1048576000
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1048576000
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 3
> scm_1 | used: 511705088
> scm_1 | readCount: 0
> scm_1 | writeCount: 32
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 511705088
> scm_1 | state: OPEN
> scm_1 | }
> scm_1 | }
> scm_1 | commandStatusReport {
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerActions {
> scm_1 | containerID: 1
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerID: 2
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | }
> scm_1 |
> scm_1 | --> RPC message response: SCMHeartbeatRequestProto to
> 172.25.0.2:33912
> scm_1 | datanodeUUID: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 |
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 1
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 1 is in CLOSING state and need not be
[jira] [Updated] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-343:
--
Attachment: HDDS-343.002.patch
> Containers are stuck in closing state in scm
>
>
> Key: HDDS-343
> URL: https://issues.apache.org/jira/browse/HDDS-343
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Fix For: 0.2.1
>
> Attachments: HDDS-343.001.patch, HDDS-343.002.patch
>
>
> Containers could not been closed currently.
> The datanode is closing the containers and sending the CLOSED state in the
> container report but SCM doesn't register that the state is closed and
> sending the close command again and again.
> I think the ContainerMapping.processContainerReport should be improved.
> {code}
> scm_1 | --> RPC message request: SCMHeartbeatRequestProto from
> 172.25.0.2:33912
> scm_1 | datanodeDetails {
> scm_1 | uuid: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 | ipAddress: "172.25.0.2"
> scm_1 | hostName: "365fd1f44f0b"
> scm_1 | ports {
> scm_1 | name: "STANDALONE"
> scm_1 | value: 9859
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "RATIS"
> scm_1 | value: 9858
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "REST"
> scm_1 | value: 9880
> scm_1 | }
> scm_1 | }
> scm_1 | nodeReport {
> scm_1 | storageReport {
> scm_1 | storageUuid: "DS-61e76107-85c5-437a-95a7-aeb8b3e7827f"
> scm_1 | storageLocation: "/tmp/hadoop-hadoop/dfs/data"
> scm_1 | capacity: 491630870528
> scm_1 | scmUsed: 2708828160
> scm_1 | remaining: 24263614464
> scm_1 | storageType: DISK
> scm_1 | failed: false
> scm_1 | }
> scm_1 | }
> scm_1 | containerReport {
> scm_1 | reports {
> scm_1 | containerID: 1
> scm_1 | used: 1061158912
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1061158912
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 2
> scm_1 | used: 1048576000
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1048576000
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 3
> scm_1 | used: 511705088
> scm_1 | readCount: 0
> scm_1 | writeCount: 32
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 511705088
> scm_1 | state: OPEN
> scm_1 | }
> scm_1 | }
> scm_1 | commandStatusReport {
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerActions {
> scm_1 | containerID: 1
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerID: 2
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | }
> scm_1 |
> scm_1 | --> RPC message response: SCMHeartbeatRequestProto to
> 172.25.0.2:33912
> scm_1 | datanodeUUID: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 |
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 1
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 1 is in CLOSING state and need not be closed.
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 2
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 2 is in CLOSING state and need not be closed.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-343:
--
Status: Patch Available (was: Open)
As I wrote I have some concerns about the existing container report handling.
Hard to find all the details as multiple classes contribute to the mail flow
(ContainerMapping, ContainerReportHandler, Node2ContainerMap). But I would keep
it in the current form for the next release.
I just added a simple step to the ContainerMapping.processContainerReport to
update the state (for that we need the proper transition event type from the
state machine).
> Containers are stuck in closing state in scm
>
>
> Key: HDDS-343
> URL: https://issues.apache.org/jira/browse/HDDS-343
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Fix For: 0.2.1
>
> Attachments: HDDS-343.001.patch
>
>
> Containers could not been closed currently.
> The datanode is closing the containers and sending the CLOSED state in the
> container report but SCM doesn't register that the state is closed and
> sending the close command again and again.
> I think the ContainerMapping.processContainerReport should be improved.
> {code}
> scm_1 | --> RPC message request: SCMHeartbeatRequestProto from
> 172.25.0.2:33912
> scm_1 | datanodeDetails {
> scm_1 | uuid: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 | ipAddress: "172.25.0.2"
> scm_1 | hostName: "365fd1f44f0b"
> scm_1 | ports {
> scm_1 | name: "STANDALONE"
> scm_1 | value: 9859
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "RATIS"
> scm_1 | value: 9858
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "REST"
> scm_1 | value: 9880
> scm_1 | }
> scm_1 | }
> scm_1 | nodeReport {
> scm_1 | storageReport {
> scm_1 | storageUuid: "DS-61e76107-85c5-437a-95a7-aeb8b3e7827f"
> scm_1 | storageLocation: "/tmp/hadoop-hadoop/dfs/data"
> scm_1 | capacity: 491630870528
> scm_1 | scmUsed: 2708828160
> scm_1 | remaining: 24263614464
> scm_1 | storageType: DISK
> scm_1 | failed: false
> scm_1 | }
> scm_1 | }
> scm_1 | containerReport {
> scm_1 | reports {
> scm_1 | containerID: 1
> scm_1 | used: 1061158912
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1061158912
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 2
> scm_1 | used: 1048576000
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1048576000
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 3
> scm_1 | used: 511705088
> scm_1 | readCount: 0
> scm_1 | writeCount: 32
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 511705088
> scm_1 | state: OPEN
> scm_1 | }
> scm_1 | }
> scm_1 | commandStatusReport {
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerActions {
> scm_1 | containerID: 1
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerID: 2
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | }
> scm_1 |
> scm_1 | --> RPC message response: SCMHeartbeatRequestProto to
> 172.25.0.2:33912
> scm_1 | datanodeUUID: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 |
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 1
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 1 is in CLOSING state and need not be closed.
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 2
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 2 is in CLOSING state and
[jira] [Assigned] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton reassigned HDDS-343:
-
Assignee: Elek, Marton
> Containers are stuck in closing state in scm
>
>
> Key: HDDS-343
> URL: https://issues.apache.org/jira/browse/HDDS-343
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Fix For: 0.2.1
>
> Attachments: HDDS-343.001.patch
>
>
> Containers could not been closed currently.
> The datanode is closing the containers and sending the CLOSED state in the
> container report but SCM doesn't register that the state is closed and
> sending the close command again and again.
> I think the ContainerMapping.processContainerReport should be improved.
> {code}
> scm_1 | --> RPC message request: SCMHeartbeatRequestProto from
> 172.25.0.2:33912
> scm_1 | datanodeDetails {
> scm_1 | uuid: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 | ipAddress: "172.25.0.2"
> scm_1 | hostName: "365fd1f44f0b"
> scm_1 | ports {
> scm_1 | name: "STANDALONE"
> scm_1 | value: 9859
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "RATIS"
> scm_1 | value: 9858
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "REST"
> scm_1 | value: 9880
> scm_1 | }
> scm_1 | }
> scm_1 | nodeReport {
> scm_1 | storageReport {
> scm_1 | storageUuid: "DS-61e76107-85c5-437a-95a7-aeb8b3e7827f"
> scm_1 | storageLocation: "/tmp/hadoop-hadoop/dfs/data"
> scm_1 | capacity: 491630870528
> scm_1 | scmUsed: 2708828160
> scm_1 | remaining: 24263614464
> scm_1 | storageType: DISK
> scm_1 | failed: false
> scm_1 | }
> scm_1 | }
> scm_1 | containerReport {
> scm_1 | reports {
> scm_1 | containerID: 1
> scm_1 | used: 1061158912
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1061158912
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 2
> scm_1 | used: 1048576000
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1048576000
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 3
> scm_1 | used: 511705088
> scm_1 | readCount: 0
> scm_1 | writeCount: 32
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 511705088
> scm_1 | state: OPEN
> scm_1 | }
> scm_1 | }
> scm_1 | commandStatusReport {
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerActions {
> scm_1 | containerID: 1
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerID: 2
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | }
> scm_1 |
> scm_1 | --> RPC message response: SCMHeartbeatRequestProto to
> 172.25.0.2:33912
> scm_1 | datanodeUUID: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 |
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 1
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 1 is in CLOSING state and need not be closed.
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 2
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 2 is in CLOSING state and need not be closed.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-343) Containers are stuck in closing state in scm
[
https://issues.apache.org/jira/browse/HDDS-343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-343:
--
Attachment: HDDS-343.001.patch
> Containers are stuck in closing state in scm
>
>
> Key: HDDS-343
> URL: https://issues.apache.org/jira/browse/HDDS-343
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Blocker
> Fix For: 0.2.1
>
> Attachments: HDDS-343.001.patch
>
>
> Containers could not been closed currently.
> The datanode is closing the containers and sending the CLOSED state in the
> container report but SCM doesn't register that the state is closed and
> sending the close command again and again.
> I think the ContainerMapping.processContainerReport should be improved.
> {code}
> scm_1 | --> RPC message request: SCMHeartbeatRequestProto from
> 172.25.0.2:33912
> scm_1 | datanodeDetails {
> scm_1 | uuid: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 | ipAddress: "172.25.0.2"
> scm_1 | hostName: "365fd1f44f0b"
> scm_1 | ports {
> scm_1 | name: "STANDALONE"
> scm_1 | value: 9859
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "RATIS"
> scm_1 | value: 9858
> scm_1 | }
> scm_1 | ports {
> scm_1 | name: "REST"
> scm_1 | value: 9880
> scm_1 | }
> scm_1 | }
> scm_1 | nodeReport {
> scm_1 | storageReport {
> scm_1 | storageUuid: "DS-61e76107-85c5-437a-95a7-aeb8b3e7827f"
> scm_1 | storageLocation: "/tmp/hadoop-hadoop/dfs/data"
> scm_1 | capacity: 491630870528
> scm_1 | scmUsed: 2708828160
> scm_1 | remaining: 24263614464
> scm_1 | storageType: DISK
> scm_1 | failed: false
> scm_1 | }
> scm_1 | }
> scm_1 | containerReport {
> scm_1 | reports {
> scm_1 | containerID: 1
> scm_1 | used: 1061158912
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1061158912
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 2
> scm_1 | used: 1048576000
> scm_1 | readCount: 0
> scm_1 | writeCount: 64
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 1048576000
> scm_1 | state: CLOSED
> scm_1 | }
> scm_1 | reports {
> scm_1 | containerID: 3
> scm_1 | used: 511705088
> scm_1 | readCount: 0
> scm_1 | writeCount: 32
> scm_1 | readBytes: 0
> scm_1 | writeBytes: 511705088
> scm_1 | state: OPEN
> scm_1 | }
> scm_1 | }
> scm_1 | commandStatusReport {
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerActions {
> scm_1 | containerID: 1
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | containerActions {
> scm_1 | containerID: 2
> scm_1 | action: CLOSE
> scm_1 | reason: CONTAINER_FULL
> scm_1 | }
> scm_1 | }
> scm_1 |
> scm_1 | --> RPC message response: SCMHeartbeatRequestProto to
> 172.25.0.2:33912
> scm_1 | datanodeUUID: "9c8f80bd-9424-4d74-99ef-a2bd58e66d7f"
> scm_1 |
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 1
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 1 is in CLOSING state and need not be closed.
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:56 -
> Close container Event triggered for container : 2
> scm_1 | 2018-08-08 16:22:51 INFO CloseContainerEventHandler:105 -
> container with id : 2 is in CLOSING state and need not be closed.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-347) Fix : testCloseContainerViaStandaAlone fails sometimes
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580909#comment-16580909
]
genericqa commented on HDDS-347:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 34m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 25s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.freon.TestFreon |
| |
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion |
| | hadoop.ozone.TestStorageContainerManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDDS-347 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935671/HDDS-347.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e66af6cc7876 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8dc07b4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| unit |
[jira] [Commented] (HDFS-13732) Erasure Coding policy name is not coming when the new policy is set
[
https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580907#comment-16580907
]
genericqa commented on HDFS-13732:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.client.impl.TestBlockReaderLocal |
| | hadoop.hdfs.TestMaintenanceState |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-13732 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935663/HDFS-13732.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux d2f846322630 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8dc07b4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24782/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDDS-336) Print out container location information for a specific ozone key
[ https://issues.apache.org/jira/browse/HDDS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580895#comment-16580895 ] Elek, Marton commented on HDDS-336: --- Thanks the interest [~GeLiXin]. 1. It's related both to RPC and Rest interface. Both of them should be improved as RpcClient and RestClient share a common interface. (But it could be separated to two different jira-s) 2. Yes containerid/localid would be enough. There is an ozone scmcli command which would give more details to a specific container id. But to use it we need the container id for a specific key. 3. It would be enough to print out these information for a specific keys not for the list of kezs. I think we can improve the RpcClient/RestClient getKeyDetails method with returning an OzoneKeyDetails instead of OzoneKey which extends the OzoneKey and contains additional information. With this approach the listKey call will be as effective as now (no additional data in the response) but the getKeyDetails could give us more detailed view which wouldn't be a significant overhead. 4. The rpc client could be easier as the rpc call already has all the required information (keyInfo.getLatestVersionLocations) only the signature of the getKeyDetails should be improved and the data returned. 5. The rest implementation may need more work as the required data should be added on the server side first. Let me know if you have additional questions. > Print out container location information for a specific ozone key > -- > > Key: HDDS-336 > URL: https://issues.apache.org/jira/browse/HDDS-336 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: SCM >Reporter: Elek, Marton >Priority: Major > Labels: newbie > Fix For: 0.2.1 > > > In the protobuf protocol we have all the containerid/localid(=blockid) > information for a specific ozone key. > It would be a big help to print out this information to the command line with > the ozone cli. > It requires to improve the REST and RPC interface with additionalOzone > KeyLocation information. > It would help a very big help during the test of the current scm behaviour. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-336) Print out container location information for a specific ozone key
[
https://issues.apache.org/jira/browse/HDDS-336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580883#comment-16580883
]
LiXin Ge commented on HDDS-336:
---
Hi [~elek], I want to take this to get familiar with the REST interface. Could
you please declare more details such like:
1. What we want in this jira is to print out the information of
containerid/localid(=blockid), or more location information from
{{datanodeDetails}} like the *fsck* command in hdfs?
2. Shall we add a option like "-location" in the existing {{ListKey}} command,
or add a new command like {{ListKeyWithLocation}}?
Thanks a lot.
> Print out container location information for a specific ozone key
> --
>
> Key: HDDS-336
> URL: https://issues.apache.org/jira/browse/HDDS-336
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: SCM
>Reporter: Elek, Marton
>Priority: Major
> Labels: newbie
> Fix For: 0.2.1
>
>
> In the protobuf protocol we have all the containerid/localid(=blockid)
> information for a specific ozone key.
> It would be a big help to print out this information to the command line with
> the ozone cli.
> It requires to improve the REST and RPC interface with additionalOzone
> KeyLocation information.
> It would help a very big help during the test of the current scm behaviour.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13217) Log audit event only used last EC policy name when add multiple policies from file
[
https://issues.apache.org/jira/browse/HDFS-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580878#comment-16580878
]
genericqa commented on HDFS-13217:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 37m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 51s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-13217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935658/HDFS-13217.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 066e2cd34dca 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8dc07b4 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24781/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24781/testReport/ |
| Max. process+thread count | 3313 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDDS-347) Fix : testCloseContainerViaStandaAlone fails sometimes
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580858#comment-16580858
]
LiXin Ge commented on HDDS-347:
---
[~xyao] Thanks for your susgestion, couldn't agree more! It's done in the 001
patch.
BTW, this change shouldn't affect the {{testCloseContainerViaRatis}} as the
v000 patch done, because the {{CloseContainer}} operation is executed
asynchronous in ratis mode and the log shown up before the state begin to
change in most cases, not like the stand_alone mode which execute it
synchronous that the log will shown up after the state changes.
> Fix : testCloseContainerViaStandaAlone fails sometimes
> --
>
> Key: HDDS-347
> URL: https://issues.apache.org/jira/browse/HDDS-347
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-347.000.patch, HDDS-347.001.patch
>
>
> This issue was finded in the automatic JenKins unit test of HDDS-265.
> The container life cycle state is : Open -> Closing -> closed, this test
> submit the container close command and wait for container state change to
> *not equal to open*, actually even when the state condition(not equal to
> open) is satisfied, the container may still in process of closing, so the LOG
> which will printf after the container closed can't be find sometimes and the
> test fails.
> {code:java|title=KeyValueContainer.java|borderStyle=solid}
> try {
> writeLock();
> containerData.closeContainer();
> File containerFile = getContainerFile();
> // update the new container data to .container File
> updateContainerFile(containerFile);
> } catch (StorageContainerException ex) {
> {code}
> Looking at the code above, the container state changes from CLOSING to CLOSED
> in the first step, the remaining *updateContainerFile* may take hundreds of
> milliseconds, so even we modify the test logic to wait for the *CLOSED* state
> will not guarantee the test success, too.
> These are two way to fix this:
> 1, Remove one of the double check which depends on the LOG.
> 2, If we have to preserve the double check, we should wait for the *CLOSED*
> state and sleep for a while to wait for the LOG appears.
> patch 000 is based on the second way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-347) Fix : testCloseContainerViaStandaAlone fails sometimes
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiXin Ge updated HDDS-347:
--
Status: Patch Available (was: Open)
> Fix : testCloseContainerViaStandaAlone fails sometimes
> --
>
> Key: HDDS-347
> URL: https://issues.apache.org/jira/browse/HDDS-347
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-347.000.patch, HDDS-347.001.patch
>
>
> This issue was finded in the automatic JenKins unit test of HDDS-265.
> The container life cycle state is : Open -> Closing -> closed, this test
> submit the container close command and wait for container state change to
> *not equal to open*, actually even when the state condition(not equal to
> open) is satisfied, the container may still in process of closing, so the LOG
> which will printf after the container closed can't be find sometimes and the
> test fails.
> {code:java|title=KeyValueContainer.java|borderStyle=solid}
> try {
> writeLock();
> containerData.closeContainer();
> File containerFile = getContainerFile();
> // update the new container data to .container File
> updateContainerFile(containerFile);
> } catch (StorageContainerException ex) {
> {code}
> Looking at the code above, the container state changes from CLOSING to CLOSED
> in the first step, the remaining *updateContainerFile* may take hundreds of
> milliseconds, so even we modify the test logic to wait for the *CLOSED* state
> will not guarantee the test success, too.
> These are two way to fix this:
> 1, Remove one of the double check which depends on the LOG.
> 2, If we have to preserve the double check, we should wait for the *CLOSED*
> state and sleep for a while to wait for the LOG appears.
> patch 000 is based on the second way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-347) Fix : testCloseContainerViaStandaAlone fails sometimes
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiXin Ge updated HDDS-347:
--
Attachment: HDDS-347.001.patch
> Fix : testCloseContainerViaStandaAlone fails sometimes
> --
>
> Key: HDDS-347
> URL: https://issues.apache.org/jira/browse/HDDS-347
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-347.000.patch, HDDS-347.001.patch
>
>
> This issue was finded in the automatic JenKins unit test of HDDS-265.
> The container life cycle state is : Open -> Closing -> closed, this test
> submit the container close command and wait for container state change to
> *not equal to open*, actually even when the state condition(not equal to
> open) is satisfied, the container may still in process of closing, so the LOG
> which will printf after the container closed can't be find sometimes and the
> test fails.
> {code:java|title=KeyValueContainer.java|borderStyle=solid}
> try {
> writeLock();
> containerData.closeContainer();
> File containerFile = getContainerFile();
> // update the new container data to .container File
> updateContainerFile(containerFile);
> } catch (StorageContainerException ex) {
> {code}
> Looking at the code above, the container state changes from CLOSING to CLOSED
> in the first step, the remaining *updateContainerFile* may take hundreds of
> milliseconds, so even we modify the test logic to wait for the *CLOSED* state
> will not guarantee the test success, too.
> These are two way to fix this:
> 1, Remove one of the double check which depends on the LOG.
> 2, If we have to preserve the double check, we should wait for the *CLOSED*
> state and sleep for a while to wait for the LOG appears.
> patch 000 is based on the second way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-342) Add example byteman script to print out hadoop rpc traffic
[ https://issues.apache.org/jira/browse/HDDS-342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580835#comment-16580835 ] Elek, Marton commented on HDDS-342: --- The problem is that by default the file is not visible from inside the docker containers. Unless we add it to the release package (but as it's a developer tool I am not sure If we should add it) or mount it manually. I created a new patch where I put the script to the dev-support (with a readm) and use the raw.githubusercontent.com url. I case github would be closed by MS it's easy to fix the url as the file is part of the hadoop source tree. > Add example byteman script to print out hadoop rpc traffic > -- > > Key: HDDS-342 > URL: https://issues.apache.org/jira/browse/HDDS-342 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > Attachments: HDDS-342.001.patch, HDDS-342.002.patch, byteman.png, > byteman2.png > > > HADOOP-15656 adds byteman support to the hadoop-runner base image. byteman is > a simple tool to define java instrumentation. For example it's very easy to > print out the incoming and outgoing hadoop rcp messages or fsimage edits. > In this patch I add one more line to the standard docker-compose cluster to > demonstrate this capability (print out rpc calls). By default it's turned off. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-342) Add example byteman script to print out hadoop rpc traffic
[ https://issues.apache.org/jira/browse/HDDS-342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-342: -- Attachment: HDDS-342.002.patch > Add example byteman script to print out hadoop rpc traffic > -- > > Key: HDDS-342 > URL: https://issues.apache.org/jira/browse/HDDS-342 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Minor > Fix For: 0.2.1 > > Attachments: HDDS-342.001.patch, HDDS-342.002.patch, byteman.png, > byteman2.png > > > HADOOP-15656 adds byteman support to the hadoop-runner base image. byteman is > a simple tool to define java instrumentation. For example it's very easy to > print out the incoming and outgoing hadoop rcp messages or fsimage edits. > In this patch I add one more line to the standard docker-compose cluster to > demonstrate this capability (print out rpc calls). By default it's turned off. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13217) Log audit event only used last EC policy name when add multiple policies from file
[
https://issues.apache.org/jira/browse/HDFS-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580816#comment-16580816
]
Kitti Nanasi commented on HDFS-13217:
-
Thanks [~liaoyuxiangqin] for fixing it!
+1 (non-binding) on patch v005
> Log audit event only used last EC policy name when add multiple policies from
> file
> ---
>
> Key: HDFS-13217
> URL: https://issues.apache.org/jira/browse/HDFS-13217
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: liaoyuxiangqin
>Assignee: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13217.001.patch, HDFS-13217.002.patch,
> HDFS-13217.003.patch, HDFS-13217.004.patch, HDFS-13217.005.patch
>
>
> When i read the addErasureCodingPolicies() of FSNamesystem class in namenode,
> i found the following code only used last ec policy name for logAuditEvent,
> i think this audit log can't track whole policies for the add multiple
> erasure coding policies to the ErasureCodingPolicyManager. Thanks.
> {code:java|title=FSNamesystem.java|borderStyle=solid}
> try {
> checkOperation(OperationCategory.WRITE);
> checkNameNodeSafeMode("Cannot add erasure coding policy");
> for (ErasureCodingPolicy policy : policies) {
> try {
> ErasureCodingPolicy newPolicy =
> FSDirErasureCodingOp.addErasureCodingPolicy(this, policy,
> logRetryCache);
> addECPolicyName = newPolicy.getName();
> responses.add(new AddErasureCodingPolicyResponse(newPolicy));
> } catch (HadoopIllegalArgumentException e) {
> responses.add(new AddErasureCodingPolicyResponse(policy, e));
> }
> }
> success = true;
> return responses.toArray(new AddErasureCodingPolicyResponse[0]);
> } finally {
> writeUnlock(operationName);
> if (success) {
> getEditLog().logSync();
> }
> logAuditEvent(success, operationName,addECPolicyName, null, null);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-347) Fix : testCloseContainerViaStandaAlone fails sometimes
[
https://issues.apache.org/jira/browse/HDDS-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiXin Ge updated HDDS-347:
--
Status: Open (was: Patch Available)
> Fix : testCloseContainerViaStandaAlone fails sometimes
> --
>
> Key: HDDS-347
> URL: https://issues.apache.org/jira/browse/HDDS-347
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: LiXin Ge
>Assignee: LiXin Ge
>Priority: Major
> Fix For: 0.2.1
>
> Attachments: HDDS-347.000.patch
>
>
> This issue was finded in the automatic JenKins unit test of HDDS-265.
> The container life cycle state is : Open -> Closing -> closed, this test
> submit the container close command and wait for container state change to
> *not equal to open*, actually even when the state condition(not equal to
> open) is satisfied, the container may still in process of closing, so the LOG
> which will printf after the container closed can't be find sometimes and the
> test fails.
> {code:java|title=KeyValueContainer.java|borderStyle=solid}
> try {
> writeLock();
> containerData.closeContainer();
> File containerFile = getContainerFile();
> // update the new container data to .container File
> updateContainerFile(containerFile);
> } catch (StorageContainerException ex) {
> {code}
> Looking at the code above, the container state changes from CLOSING to CLOSED
> in the first step, the remaining *updateContainerFile* may take hundreds of
> milliseconds, so even we modify the test logic to wait for the *CLOSED* state
> will not guarantee the test success, too.
> These are two way to fix this:
> 1, Remove one of the double check which depends on the LOG.
> 2, If we have to preserve the double check, we should wait for the *CLOSED*
> state and sleep for a while to wait for the LOG appears.
> patch 000 is based on the second way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13732) Erasure Coding policy name is not coming when the new policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zsolt Venczel updated HDFS-13732: - Status: Patch Available (was: In Progress) > Erasure Coding policy name is not coming when the new policy is set > --- > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13732) Erasure Coding policy name is not coming when the new policy is set
[ https://issues.apache.org/jira/browse/HDFS-13732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zsolt Venczel updated HDFS-13732: - Attachment: HDFS-13732.01.patch > Erasure Coding policy name is not coming when the new policy is set > --- > > Key: HDFS-13732 > URL: https://issues.apache.org/jira/browse/HDFS-13732 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, tools >Affects Versions: 3.0.0 >Reporter: Soumyapn >Assignee: Zsolt Venczel >Priority: Trivial > Attachments: EC_Policy.PNG, HDFS-13732.01.patch > > > Scenerio: > If the new policy apart from the default EC policy is set for the HDFS > directory, then the console message is coming as "Set default erasure coding > policy on " > Expected output: > It would be good If the EC policy name is displayed when the policy is set... > > Actual output: > Set default erasure coding policy on > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12862) CacheDirective may invalidata,when NN restart or make a transition to Active.
[
https://issues.apache.org/jira/browse/HDFS-12862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580794#comment-16580794
]
genericqa commented on HDFS-12862:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}164m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | HDFS-12862 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935653/HDFS-12862-trunk.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 30072ea31f4f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / bdd0e01 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24780/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24780/testReport/ |
| Max. process+thread count | 3741 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24780/console |
