[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[
https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737428#comment-16737428
]
Hadoop QA commented on HDFS-13312:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-13312 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13312 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921423/HDFS-13312.003.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25932/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> NameNode High Availability ZooKeeper based discovery rather than explicit
> nn1,nn2 configs
> -
>
> Key: HDFS-13312
> URL: https://issues.apache.org/jira/browse/HDFS-13312
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, hdfs, namenode, nn
>Affects Versions: 2.9.1
>Reporter: Hari Sekhon
>Priority: Major
> Attachments: HDFS-13312.000.patch, HDFS-13312.001.patch,
> HDFS-13312.002.patch, HDFS-13312.003.patch
>
>
> Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic
> discovery rather than explicitly setting the NameNode addresses via nn1,n2 in
> the configs.
> One proprietary Hadoop vendor already uses ZK for Resource Manager HA
> discovery - it makes sense that the open source core should do this for both
> Yarn and HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[
https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621027#comment-16621027
]
Hadoop QA commented on HDFS-13312:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-13312 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13312 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921423/HDFS-13312.003.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/25101/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> NameNode High Availability ZooKeeper based discovery rather than explicit
> nn1,nn2 configs
> -
>
> Key: HDFS-13312
> URL: https://issues.apache.org/jira/browse/HDFS-13312
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, hdfs, namenode, nn
>Affects Versions: 2.9.1
>Reporter: Hari Sekhon
>Priority: Major
> Attachments: HDFS-13312.000.patch, HDFS-13312.001.patch,
> HDFS-13312.002.patch, HDFS-13312.003.patch
>
>
> Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic
> discovery rather than explicitly setting the NameNode addresses via nn1,n2 in
> the configs.
> One proprietary Hadoop vendor already uses ZK for Resource Manager HA
> discovery - it makes sense that the open source core should do this for both
> Yarn and HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[ https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621015#comment-16621015 ] Íñigo Goiri commented on HDFS-13312: As I'm working on running Hadoop on VMs, I see more and more the need to have a framework to discover servers. I played with DNS but this works for fixed setups but it's not very elastic (I'm limited to having always say 3 NNs). I'll open a JIRA in commons to define the framework and use it as the branch for working on this. As there are subparts in both HDFS and YARN this won't be a pure umbrella; I hope linking JIRAs and merging into the branch is fine. Anyway, I'll link the JIRA once I open it here and we can start the discussion there. > NameNode High Availability ZooKeeper based discovery rather than explicit > nn1,nn2 configs > - > > Key: HDFS-13312 > URL: https://issues.apache.org/jira/browse/HDFS-13312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs, namenode, nn >Affects Versions: 2.9.1 >Reporter: Hari Sekhon >Priority: Major > Attachments: HDFS-13312.000.patch, HDFS-13312.001.patch, > HDFS-13312.002.patch, HDFS-13312.003.patch > > > Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic > discovery rather than explicitly setting the NameNode addresses via nn1,n2 in > the configs. > One proprietary Hadoop vendor already uses ZK for Resource Manager HA > discovery - it makes sense that the open source core should do this for both > Yarn and HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[ https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475015#comment-16475015 ] Íñigo Goiri commented on HDFS-13312: To be fair, given the lack of agreement, I'm not even sure is worth moving forward with this at all. > NameNode High Availability ZooKeeper based discovery rather than explicit > nn1,nn2 configs > - > > Key: HDFS-13312 > URL: https://issues.apache.org/jira/browse/HDFS-13312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs, namenode, nn >Affects Versions: 2.9.1 >Reporter: Hari Sekhon >Priority: Major > Attachments: HDFS-13312.000.patch, HDFS-13312.001.patch, > HDFS-13312.002.patch, HDFS-13312.003.patch > > > Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic > discovery rather than explicitly setting the NameNode addresses via nn1,n2 in > the configs. > One proprietary Hadoop vendor already uses ZK for Resource Manager HA > discovery - it makes sense that the open source core should do this for both > Yarn and HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[ https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474688#comment-16474688 ] Anu Engineer commented on HDFS-13312: - As I commented in HDFS-13098, any fundamental changes to HDFS protocol should be done in a branch. Thx, > NameNode High Availability ZooKeeper based discovery rather than explicit > nn1,nn2 configs > - > > Key: HDFS-13312 > URL: https://issues.apache.org/jira/browse/HDFS-13312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs, namenode, nn >Affects Versions: 2.9.1 >Reporter: Hari Sekhon >Priority: Major > Attachments: HDFS-13312.000.patch, HDFS-13312.001.patch, > HDFS-13312.002.patch, HDFS-13312.003.patch > > > Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic > discovery rather than explicitly setting the NameNode addresses via nn1,n2 in > the configs. > One proprietary Hadoop vendor already uses ZK for Resource Manager HA > discovery - it makes sense that the open source core should do this for both > Yarn and HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[
https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460175#comment-16460175
]
genericqa commented on HDFS-13312:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
3s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 48s{color} | {color:orange} root: The patch generated 13 new + 494 unchanged
- 0 fixed = 507 total (was 494) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 14s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
29s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-client generated 2 new
+ 0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
46s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}234m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| |
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[
https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459933#comment-16459933
]
Íñigo Goiri commented on HDFS-13312:
[~daryn] thanks for the comments, let me split this into low hanging fruit
fixes and larger holistic issues.
bq. The discovery calls now all catch IOE, log, and return null. That needs to
be removed.
Done, I kept the same exception API as the original
{{DFSUtil}}/{{DFSUtilClient}} ones. Those could still use some tweaking but
let's keep what it's there.
{quote}
* The discovery cache is not thread safe.
* It's not safe to use Configuration#hashCode as a cache key. Hash codes are
not guaranteed to be unique.
{quote}
I moved the whole caching into CacheLoader so it takes care of thread safety.
I also made the key to support both hash and equals so it should be fine.
Still not 100% sure if Configuration#equals is doing the proper comparisson but
it looks like
bq. DynamicNameserviceDiscovery only exists to auto-magically use
ConfiguredFailoverProxyProvider as a conf default. Is that really needed?
Correct. I was trying to get a better way to figure it out but not sure how.
Any other way of triggering this?
Regarding the higher level questions:
bq. That's a excellent argument for not including the ZK implementation.
I will personally not use it but I see a couple reasons for this implementation:
* There are requests for having this; [~harisekhon] opened this JIRA as an
example.
* It sets a good base for starting this dynamic discovery without breaking
compatibility.
bq. However replacing confs with another conf key is equally as flawed.
I'm not sure the argument of this being another key is fair. The only reason
that this has a configuration is to make pluggable.
bq. The solution I've floated is use DNS.
In general I like that but I see two issues:
* It's a big change that would span a lot of components. We would need a new
umbrella for that; we can start the discussion there and go deeper into what
this would look like.
* I'm not sure if you are moving the issue into the configuration of the DNS;
for example, it doesn't look easy to define shared/non-shared subclusters in
the network.
> NameNode High Availability ZooKeeper based discovery rather than explicit
> nn1,nn2 configs
> -
>
> Key: HDFS-13312
> URL: https://issues.apache.org/jira/browse/HDFS-13312
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, hdfs, namenode, nn
>Affects Versions: 2.9.1
>Reporter: Hari Sekhon
>Priority: Major
> Attachments: HDFS-13312.000.patch, HDFS-13312.001.patch,
> HDFS-13312.002.patch, HDFS-13312.003.patch
>
>
> Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic
> discovery rather than explicitly setting the NameNode addresses via nn1,n2 in
> the configs.
> One proprietary Hadoop vendor already uses ZK for Resource Manager HA
> discovery - it makes sense that the open source core should do this for both
> Yarn and HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[
https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459764#comment-16459764
]
Daryn Sharp commented on HDFS-13312:
bq. Yes, I don't expect this to work well for more than a couple tens of
clients/DNs.
That's a excellent argument for not including the ZK implementation.
More specific to the patch:
* The discovery calls now all catch IOE, log, and return null. That needs to
be removed.
* The discovery cache is not thread safe.
* It's not safe to use {{Configuration#hashCode}} as a cache key. Hash codes
are not guaranteed to be unique.
* {{DynamicNameserviceDiscovery}} only exists to auto-magically use
{{ConfiguredFailoverProxyProvider}} as a conf default. Is that really needed?
I've always considered config file based HA to be a horrible broken design, so
I'm glad you are working to change it. However replacing confs with another
conf key is equally as flawed. The logistics of managing the updates of user
confs is very difficult. Moving the discovery into the router is an equally
bad idea when there are clusters other than those served by the router.
––
The solution I've floated is use DNS. Distributed discovery is a solved
problem. Now we could get rid of the HA vs non-HA proxy providers. Resolve
all addresses for a host. Use one proxy provider that instantiates clients for
all those addresses. Failover just roves the array. It doesn't matter if the
array has 1 or 2 hosts which will work great for IP failover (since the NN
appears to be a single host) and we can remove our custom proxy class.
> NameNode High Availability ZooKeeper based discovery rather than explicit
> nn1,nn2 configs
> -
>
> Key: HDFS-13312
> URL: https://issues.apache.org/jira/browse/HDFS-13312
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, hdfs, namenode, nn
>Affects Versions: 2.9.1
>Reporter: Hari Sekhon
>Priority: Major
> Attachments: HDFS-13312.000.patch, HDFS-13312.001.patch,
> HDFS-13312.002.patch
>
>
> Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic
> discovery rather than explicitly setting the NameNode addresses via nn1,n2 in
> the configs.
> One proprietary Hadoop vendor already uses ZK for Resource Manager HA
> discovery - it makes sense that the open source core should do this for both
> Yarn and HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[
https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459499#comment-16459499
]
genericqa commented on HDFS-13312:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 31m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 30m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 15s{color} | {color:orange} root: The patch generated 13 new + 494 unchanged
- 0 fixed = 507 total (was 494) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 10m
2s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
38s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-client generated 2 new
+ 0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
51s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestParallelShortCircuitRead |
| |
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[ https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459426#comment-16459426 ] Íñigo Goiri commented on HDFS-13312: In [^HDFS-13312.002.patch], I added an approach to get the failover provider by default and if it the service is supposed to be dynamic, it will use the default ConfiguredFailoverProxyProvider. > NameNode High Availability ZooKeeper based discovery rather than explicit > nn1,nn2 configs > - > > Key: HDFS-13312 > URL: https://issues.apache.org/jira/browse/HDFS-13312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs, namenode, nn >Affects Versions: 2.9.1 >Reporter: Hari Sekhon >Priority: Major > Attachments: HDFS-13312.000.patch, HDFS-13312.001.patch, > HDFS-13312.002.patch > > > Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic > discovery rather than explicitly setting the NameNode addresses via nn1,n2 in > the configs. > One proprietary Hadoop vendor already uses ZK for Resource Manager HA > discovery - it makes sense that the open source core should do this for both > Yarn and HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[
https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459382#comment-16459382
]
genericqa commented on HDFS-13312:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
4m 29s{color} | {color:orange} root: The patch generated 13 new + 488 unchanged
- 0 fixed = 501 total (was 488) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
42s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}222m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| |
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[ https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459169#comment-16459169 ] Íñigo Goiri commented on HDFS-13312: bq. You might want to talk to ZK devs. An internal dev says ZK will not scale to support this use case. Yes, I don't expect this to work well for more than a couple tens of clients/DNs. This is for simple cases and setting up the framework to make it pluggable. The solution we will use internally will be based on the Routers which will allow it to scale and make it more dynamic. I added [^HDFS-13312.001.patch] which in addition to the DNs, it also allows clients to discover the addresses and connect to them. I had to split the discoveyr service for the client and the server sides and it's not as clean as I'd like but does the job. > NameNode High Availability ZooKeeper based discovery rather than explicit > nn1,nn2 configs > - > > Key: HDFS-13312 > URL: https://issues.apache.org/jira/browse/HDFS-13312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs, namenode, nn >Affects Versions: 2.9.1 >Reporter: Hari Sekhon >Priority: Major > Attachments: HDFS-13312.000.patch, HDFS-13312.001.patch > > > Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic > discovery rather than explicitly setting the NameNode addresses via nn1,n2 in > the configs. > One proprietary Hadoop vendor already uses ZK for Resource Manager HA > discovery - it makes sense that the open source core should do this for both > Yarn and HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[ https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459021#comment-16459021 ] Daryn Sharp commented on HDFS-13312: You might want to talk to ZK devs. An internal dev says ZK will not scale to support this use case. I suppose it's pluggable and won't hurt me though... > NameNode High Availability ZooKeeper based discovery rather than explicit > nn1,nn2 configs > - > > Key: HDFS-13312 > URL: https://issues.apache.org/jira/browse/HDFS-13312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs, namenode, nn >Affects Versions: 2.9.1 >Reporter: Hari Sekhon >Priority: Major > Attachments: HDFS-13312.000.patch > > > Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic > discovery rather than explicitly setting the NameNode addresses via nn1,n2 in > the configs. > One proprietary Hadoop vendor already uses ZK for Resource Manager HA > discovery - it makes sense that the open source core should do this for both > Yarn and HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[
https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458264#comment-16458264
]
genericqa commented on HDFS-13312:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 438 unchanged - 0 fixed = 442 total (was 438) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.client.impl.TestBlockReaderLocal |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSClientRetries |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13312 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921196/HDFS-13312.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 268bd4a40ad6 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / eb7fe1d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
|
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[ https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458222#comment-16458222 ] Íñigo Goiri commented on HDFS-13312: As a disclaimer, in the future I will add a RouterBasedNameserviceDiscovery which will leverage the Router from RBF to do this discovery. That will allow us to define fancier policies and defining subcluster on the fly. That will be part of HDFS-13098 once we have this new pluggable approach defined. > NameNode High Availability ZooKeeper based discovery rather than explicit > nn1,nn2 configs > - > > Key: HDFS-13312 > URL: https://issues.apache.org/jira/browse/HDFS-13312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs, namenode, nn >Affects Versions: 2.9.1 >Reporter: Hari Sekhon >Priority: Major > Attachments: HDFS-13312.000.patch > > > Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic > discovery rather than explicitly setting the NameNode addresses via nn1,n2 in > the configs. > One proprietary Hadoop vendor already uses ZK for Resource Manager HA > discovery - it makes sense that the open source core should do this for both > Yarn and HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[
https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458220#comment-16458220
]
Íñigo Goiri commented on HDFS-13312:
The Datanode itself is already pretty well done for dynamically adding/removing
nameservices.
I made the discovery a pluggable module.
The current configuration is still implemented in
{{ConfigurationBasedNameserviceDiscovery}}.
I also added a new approach based on ZooKeeper in
{{ZookeeperBasedNameserviceDiscovery}}.
Please, take a look at [^HDFS-13312.000.patch].
If this approach suffices, I would add unit tests and cleanup the ZK
implementation as it is pretty basic at this point.
> NameNode High Availability ZooKeeper based discovery rather than explicit
> nn1,nn2 configs
> -
>
> Key: HDFS-13312
> URL: https://issues.apache.org/jira/browse/HDFS-13312
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: ha, hdfs, namenode, nn
>Affects Versions: 2.9.1
>Reporter: Hari Sekhon
>Priority: Major
> Attachments: HDFS-13312.000.patch
>
>
> Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic
> discovery rather than explicitly setting the NameNode addresses via nn1,n2 in
> the configs.
> One proprietary Hadoop vendor already uses ZK for Resource Manager HA
> discovery - it makes sense that the open source core should do this for both
> Yarn and HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13312) NameNode High Availability ZooKeeper based discovery rather than explicit nn1,nn2 configs
[ https://issues.apache.org/jira/browse/HDFS-13312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408818#comment-16408818 ] Íñigo Goiri commented on HDFS-13312: In HDFS-13098, we would like to support the Datanodes contacting the Routers to discover the NNs instead of explicitly setting up the nameservice. I think that the discovery described in this JIRA may target the clients and not the workers (i.e., DNs). In any case, I think adding this support is valuable and HDFS-13098 could leverage it too (similar for YARN). Can you point to the solution to do RM HA discovery? > NameNode High Availability ZooKeeper based discovery rather than explicit > nn1,nn2 configs > - > > Key: HDFS-13312 > URL: https://issues.apache.org/jira/browse/HDFS-13312 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, hdfs, namenode, nn >Affects Versions: 2.9.1 >Reporter: Hari Sekhon >Priority: Major > > Improvement Request for HDFS NameNode HA to use ZooKeeper based dynamic > discovery rather than explicitly setting the NameNode addresses via nn1,n2 in > the configs. > One proprietary Hadoop vendor already uses ZK for Resource Manager HA > discovery - it makes sense that the open source core should do this for both > Yarn and HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
