[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[ https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396408#comment-16396408 ] Konstantin Shvachko commented on HDFS-12975: Reviewed 003 patch. Looks good. +1. I think we should be able to transition standby to observer and back. But active should be able to transition only to standby. At least initially, we may "optimize" that later. Will update the design doc with this info. > Changes to the NameNode to support reads from standby > - > > Key: HDFS-12975 > URL: https://issues.apache.org/jira/browse/HDFS-12975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-12975.000.patch, HDFS-12975.001.patch, > HDFS-12975.002.patch, HDFS-12975.003.patch > > > In order to support reads from standby NameNode needs changes to add Observer > role, turn off checkpointing and such. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387467#comment-16387467
]
genericqa commented on HDFS-12975:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
54s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}148m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}198m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.federation.router.TestRouterSafemode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-12975 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913153/HDFS-12975.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 877bb6a3d317 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 745190e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[ https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387309#comment-16387309 ] Chao Sun commented on HDFS-12975: - Thanks Konstantin. Yes I agree that option 2) seems not necessary. Regarding transitions, do you think we should support bi-directional transition between standby and observer? how about transition between active and observer? Also, with this setup I'm not sure whether we need the automatic failover from observer to active, in case standby is not available. IMHO, most cases one active and one standby works pretty well, so as long as we guarantee there's at least one standby it should be OK? Also thanks for the comments. Will upload a new patch soon. > Changes to the NameNode to support reads from standby > - > > Key: HDFS-12975 > URL: https://issues.apache.org/jira/browse/HDFS-12975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-12975.000.patch, HDFS-12975.001.patch, > HDFS-12975.002.patch, HDFS-12975.003.patch > > > In order to support reads from standby NameNode needs changes to add Observer > role, turn off checkpointing and such. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[ https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384919#comment-16384919 ] Konstantin Shvachko commented on HDFS-12975: Also was thinking about the startup sequence. I think it wouldn't be right to use a configuration variable to identify, which nodes are observers. NameNodes can transition back and force in run-time, while the config file will not be up to date with these changes. So we should stick to option (1) above that is explicitly transition standbys to observers via an admin command. > Changes to the NameNode to support reads from standby > - > > Key: HDFS-12975 > URL: https://issues.apache.org/jira/browse/HDFS-12975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-12975.000.patch, HDFS-12975.001.patch, > HDFS-12975.002.patch > > > In order to support reads from standby NameNode needs changes to add Observer > role, turn off checkpointing and such. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384909#comment-16384909
]
Konstantin Shvachko commented on HDFS-12975:
Patch 002 looks good. There a few white space changes. And I have one
suggestion: instead of
{code}
!nn.isStandbyState() && !nn.isObserverState()
{code}
we can use
{code}
nn.isActiveState()
{code}
which is actually the intended logic worth changing in trunk.
> Changes to the NameNode to support reads from standby
> -
>
> Key: HDFS-12975
> URL: https://issues.apache.org/jira/browse/HDFS-12975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-12975.000.patch, HDFS-12975.001.patch,
> HDFS-12975.002.patch
>
>
> In order to support reads from standby NameNode needs changes to add Observer
> role, turn off checkpointing and such.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16380068#comment-16380068
]
genericqa commented on HDFS-12975:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12975 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912394/HDFS-12975.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0f2264fa1897 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a9c14b1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23237/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[ https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379882#comment-16379882 ] Chao Sun commented on HDFS-12975: - Attached patch v2. > Changes to the NameNode to support reads from standby > - > > Key: HDFS-12975 > URL: https://issues.apache.org/jira/browse/HDFS-12975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-12975.000.patch, HDFS-12975.001.patch, > HDFS-12975.002.patch > > > In order to support reads from standby NameNode needs changes to add Observer > role, turn off checkpointing and such. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379865#comment-16379865
]
Chao Sun commented on HDFS-12975:
-
Thanks Konstantin. Let me strip out the state part of the patch and leave the
rest for separate Jiras.
{quote}
May be we should have a separate doc or I can update the design. The main thing
is that we should configure NNs altogether the same way as it is currently done
{quote}
Yes, I think it would be useful to add a section for the configuration changes.
Currently I'm using a separate config:
{{dfs.ha.observer.namenode.}}, to separate observer from
active/standby. I think this is similar to the choices #2 you mentioned.
{quote}
HAUtil.getConfForOtherNodes() the refactoring (otherNn -> otherNameNodes)
should go into trunk. Better avoid renaming even though the original choice of
the variable name is bad.
{quote}
Will create a Jira to address this.
> Changes to the NameNode to support reads from standby
> -
>
> Key: HDFS-12975
> URL: https://issues.apache.org/jira/browse/HDFS-12975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-12975.000.patch, HDFS-12975.001.patch
>
>
> In order to support reads from standby NameNode needs changes to add Observer
> role, turn off checkpointing and such.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379616#comment-16379616
]
Konstantin Shvachko commented on HDFS-12975:
Chao, the state part of the patch looks really good to me. I think we can
commit it if you can separate it from the rest, after some minor cleanup - see
below.
*_Configuration needs more thinking_.* I missed to describe the cluster startup
process in the design doc. May be we should have a separate doc or I can update
the design. The main thing is that we should configure NNs altogether the same
way as it is currently done:
{code:java}
dfs.ha.namenodes.mycluster = nn1,nn2,nn3
dfs.namenode.rpc-address.mycluster.nn1 = machine1.example.com:8020
dfs.namenode.rpc-address.mycluster.nn2 = machine2.example.com:8020
dfs.namenode.rpc-address.mycluster.nn3 = machine3.example.com:8020
{code}
Currently all nodes start in standby state and admins should transition to
active one of them, e.g. nn1.
We have two choices for observers.
# Use {{haadmin transitionToObserver}}, which will bring say nn3 from standby
to observer state.
# Or add configuration parameter, which allows nn3 to start in observer state
{code:java}
dfs.namenode.observer.mycluster.nn3 = true // false by default
{code}
In the end we will probably need both.
_*Comments on the rest fo the patch*_
# {{allowStaleStandbyReads}} for observers should be true - ignore the config
# NameNodeStatusMXBean methods should not be in this patch
** {{NameNode.getRole()}} looks like incompatible change
** {{NameNode.getState()}} shouldn't change. You need {{synchronized}} to
access the {{state}} member.
# {{HAState}} adds only an unused import
# {{HAUtil.getConfForOtherNodes()}} the refactoring (otherNn ->
otherNameNodes) should go into trunk. Better avoid renaming even though the
original choice of the variable name is bad.
# {{StandbyState.checkOperation()}} should not change since
{{allowStaleReads()}} for observer is true
# A think Command {{-namenodes}} should return all namenodes including
standbys and observers. As it does now. We can postpone this decision and do it
in the next jira if you want.
# Even in tests we should not count observers separately from NNs. They are
all NNs.
# I see some Javadoc spelling corrections, white spaces, and long lines
** If formatting or spelling corrections of the existing code are needed they
should go into trunk
** New ones should be fixed in the
> Changes to the NameNode to support reads from standby
> -
>
> Key: HDFS-12975
> URL: https://issues.apache.org/jira/browse/HDFS-12975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-12975.000.patch, HDFS-12975.001.patch
>
>
> In order to support reads from standby NameNode needs changes to add Observer
> role, turn off checkpointing and such.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374118#comment-16374118
]
genericqa commented on HDFS-12975:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-hdfs-project: The patch generated 68 new
+ 1270 unchanged - 15 fixed = 1338 total (was 1285) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 38s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
21s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}135m 39s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}192m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12975 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911677/HDFS-12975.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 208d0ee1537d 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision |
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[ https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373919#comment-16373919 ] Chao Sun commented on HDFS-12975: - OK, sounds good. I'll post a patch soon without the failover change. Will also create a separate Jira for the issue. > Changes to the NameNode to support reads from standby > - > > Key: HDFS-12975 > URL: https://issues.apache.org/jira/browse/HDFS-12975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-12975.000.patch > > > In order to support reads from standby NameNode needs changes to add Observer > role, turn off checkpointing and such. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[ https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373822#comment-16373822 ] Konstantin Shvachko commented on HDFS-12975: Hey [~csun] I don't think we should target ZK failover for this jira. If manual failover works to any node it is great. But even that is not a requirement for this issue. I would suggest to handle HA issues in a separate jira. We should gradually introduce new functionality with smaller patches. If a change is too big split it into smaller work items. > Changes to the NameNode to support reads from standby > - > > Key: HDFS-12975 > URL: https://issues.apache.org/jira/browse/HDFS-12975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-12975.000.patch > > > In order to support reads from standby NameNode needs changes to add Observer > role, turn off checkpointing and such. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373538#comment-16373538
]
Chao Sun commented on HDFS-12975:
-
Thanks [~shv] for taking a look! I've fixed the changes for the 1) and 3), and
currently is working on a way to let observer participate in failover. There
are some complexities:
According to the design, during failover we want to choose SBN over Observer if
the former is available. Therefore, the logic in {{ZKFailoverController}} will
need to change.
First, for graceful failover, I'm thinking about associating each target with a
flag indicating whether it is a "preferred" target (e.g., SBN).
In {{ZKFailoverController::doGracefulFailover()}} we will then check the flag
of the local target against all the flags of the other nodes, and determine the
action (e.g., when local target is non-preferred and there exists at least one
preferred target in other nodes, quit the election).
Second, for the case of "ungraceful" failover, such as ANN crash, it seems more
difficult. The current mechanism is to have all the available candidates to
compete for the ZK lock, so there is no way for them to communicate and
coordinate such that SBN will be preferred.
I'll continue to look into this. But let me know if you have any good
suggestions.
> Changes to the NameNode to support reads from standby
> -
>
> Key: HDFS-12975
> URL: https://issues.apache.org/jira/browse/HDFS-12975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-12975.000.patch
>
>
> In order to support reads from standby NameNode needs changes to add Observer
> role, turn off checkpointing and such.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366438#comment-16366438
]
Konstantin Shvachko commented on HDFS-12975:
Hey [~csun], was looking through your patch.
# There are some changes that are not related to incorporating Observer
capabilities into NameNode, like start/stop scripts, {{ReadOnly}} annotations,
{{hdfs-default.xml}}. We should introduce them in separate jira(s). Would be
good to dedicate this mostly the NameNode changes.
# The main question in this regard is whether we define observer role as a
separate class {{ObserverState}} as in your patch or as an attribute (flag) of
the {{StandbyState}}. I see that for example {{startObserverState()}} reuses a
lot of code of {{startStandbyServices()}}. Also in the end ANN should be able
to fail over to either SBN or Observer, so may be we don't need a new state.
Or add it when we really have to.
# White space changes are mostly annoying, but also make it hard to merge /
rebase the branch in the future, which we will have to do a lot.
> Changes to the NameNode to support reads from standby
> -
>
> Key: HDFS-12975
> URL: https://issues.apache.org/jira/browse/HDFS-12975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Konstantin Shvachko
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-12975.000.patch
>
>
> In order to support reads from standby NameNode needs changes to add Observer
> role, turn off checkpointing and such.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357776#comment-16357776
]
genericqa commented on HDFS-12975:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
29s{color} | {color:red} root in trunk failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
28s{color} | {color:red} hadoop-hdfs-project in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
39s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
31s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 32s{color}
| {color:red} hadoop-hdfs-project generated 431 new + 0 unchanged - 0 fixed =
431 total (was 0) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 56s{color} | {color:orange} hadoop-hdfs-project: The patch generated 926 new
+ 0 unchanged - 0 fixed = 926 total (was 0) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
2s{color} | {color:red} The patch generated 2 new + 10 unchanged - 0 fixed = 12
total (was 10) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
54s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 4 new + 1
unchanged - 0 fixed = 5 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
25s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[ https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357734#comment-16357734 ] Chao Sun commented on HDFS-12975: - Attached a working patch just to trigger jenkins and check whether there's any test regression. It's not ready yet. > Changes to the NameNode to support reads from standby > - > > Key: HDFS-12975 > URL: https://issues.apache.org/jira/browse/HDFS-12975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Konstantin Shvachko >Assignee: Chao Sun >Priority: Major > Attachments: HDFS-12975.000.patch > > > In order to support reads from standby NameNode needs changes to add Observer > role, turn off checkpointing and such. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
