[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16410438#comment-16410438 ] Hudson commented on HDFS-10441: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10441: libhdfs++: HA namenode support. Contributed by James (james.clampffer: rev 4cb0dad5e535f4ee79c32a54f8958bf7d4402123) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/uri.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/status.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/namenode_info.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/retry_policy.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/namenode_info.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/uri.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/retry_policy.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/options.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/options.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/events.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.h > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-10441.HDFS-8707.010.patch, HDFS-10441.HDFS-8707.011.patch, > HDFS-10441.HDFS-8707.012.patch, HDFS-10441.HDFS-8707.013.patch, > HDFS-10441.HDFS-8707.014.patch, HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397547#comment-15397547 ] Bob Hansen commented on HDFS-10441: --- James - thanks for the responses. +1 > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-10441.HDFS-8707.010.patch, HDFS-10441.HDFS-8707.011.patch, > HDFS-10441.HDFS-8707.012.patch, HDFS-10441.HDFS-8707.013.patch, > HDFS-10441.HDFS-8707.014.patch, HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396472#comment-15396472 ] James Clampffer commented on HDFS-10441: Thanks for looking at this more Bob. bq. Should HandleRpcResponse return a value or just forward-propagate to CommsError? In the implementation, we alternately do one, the other, or both. Probably best write functions as plain procedural code that return values wherever possible IMO. That way the async work and associated complexity are clustered in the same areas rather than distributed across the code. It might be worth splitting off a seperate jira to refactor and document the RPC stuff; the current mix of state machine(s) and continuations was a little tricky to reverse engineer for adding this feature. As you pointed out I'm doing both returns and forwarding here. The current code is fairly well tested/used and I'd really like to get a CI test around this before changing it too much more. Think that's worth a seperate jira? bq. RpcConnectionImpl::ConnectAndFlush logs entry at the INFO level. I think we already log once when we're attempting to connect, do we not? If this is the only place we keep it, we should include the endpoint we're connecting to in the log message. Yea, RpcConnectionImpl::ConnectAndFlush and RpcConnectionImpl::Connect are both logging effectively the same thing at the moment. Logging both helps a little bit to reason about the indirect recursion via async callbacks going on in that code. I think it'd be best to wait until the connection callback to log the endpoint in case there are multiple endpoints, but that starts getting a bit out of scope for this. My general strategy is to follow "perfect is the enemy of good" when it comes to this sort of stuff, particularly early in the lifetime of a feature. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-10441.HDFS-8707.010.patch, HDFS-10441.HDFS-8707.011.patch, > HDFS-10441.HDFS-8707.012.patch, HDFS-10441.HDFS-8707.013.patch, > HDFS-10441.HDFS-8707.014.patch, HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391737#comment-15391737 ] Bob Hansen commented on HDFS-10441: --- A few minor points (not necessarily worth holding up this patch): Should HandleRpcResponse return a value or just forward-propagate to CommsError? In the implementation, we alternately do one, the other, or both. RpcConnectionImpl::ConnectAndFlush logs entry at the INFO level. I think we already log once when we're attempting to connect, do we not? If this is the only place we keep it, we should include the endpoint we're connecting to in the log message. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-10441.HDFS-8707.010.patch, HDFS-10441.HDFS-8707.011.patch, > HDFS-10441.HDFS-8707.012.patch, HDFS-10441.HDFS-8707.013.patch, > HDFS-10441.HDFS-8707.014.patch, HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380267#comment-15380267 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 57s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 38s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 13s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 13s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 24s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 26s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 34s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 58m 50s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12818244/HDFS-10441.HDFS-8707.014.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux 7bb87c634a8c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / d18e396 | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16072/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16072/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379889#comment-15379889 ] James Clampffer commented on HDFS-10441: bq. In RpcConnectionImpl::OnRecvCompleted, if we detect that we've connected to the standby, it falls through to StartReading(). Should it bail out at that point? I tried this, but if we bail out here we need something else to get the rpc loop to start running again. It seemed like this was a relatively simple way of solving that rather than adding a special case. I could be missing somethere here, I had a sketch of the Rpc code modeled as a state machine but it's possible that it is out of date now. In RpcEngine::RpcCommsError, we call pendingRequests [ i ]->IncrementFailoverCount(); should that implicitly reset the retry count to 0? Will we get into cases where it retries until it fails, then the retry count is already == max_retry? Nice catch. I had the failover case covered: {code} head_action = RetryAction::failover(std::max(0,options_.rpc_retry_delay_ms)); {code} I'll move that into IncrementRetryCount to cover all cases. bq. If a namenode is down when we try to resolve, we don't try again when it's time to fail over, do we? We should capture that in another bug We do at the bottom of HANamenodeTracker::GetFailoverAndUpdate when we call ResolveInPlace. The idea is that the endpoint vector will be empty either because it's unset or explicitly cleared when resolution fails so we can just do "if empty ResolveInPlace". future discussion bq. In FixedDealyWithFailover::ShouldRetry(), should we failover on any other errors other than timeout? Bad route to host? DNS failure? I have to check the java code to be 100% sure. Based on the user configuration options it looked like timeout was the main one that needed to be accounted for. Bad route to host should probably fall under that rule as well since it doesn't seem like it can be recovered from. With a DNS failure we might be out of luck in general, but it might be worth propagating back to the user. This is just the quick failover path, everything will end up failing over but will retry first. bq. In FixedDealyWithFailover::ShouldRetry(), we're always using a delay if retries < 3. This should be configurable. We can cover that in another bug Oh, I'm not sure how I missed that. Should be using FixedDelayWithFailover::max_retry_. That said there's still work to be done on the delay logic to get parity with the Java client notably things like exponential backoff. I left that out until good tests are added because it started adding more corner cases that I had to test manually and because the simple workaround is to bump up max failovers. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-10441.HDFS-8707.010.patch, HDFS-10441.HDFS-8707.011.patch, > HDFS-10441.HDFS-8707.012.patch, HDFS-10441.HDFS-8707.013.patch, > HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15379522#comment-15379522 ] Bob Hansen commented on HDFS-10441: --- Just a _few_ more questions: * In RpcConnectionImpl::OnRecvCompleted, if we detect that we've connected to the standby, it falls through to StartReading(). Should it bail out at that point? * In RpcEngine::RpcCommsError, we call pendingRequests[i]->IncrementFailoverCount(); should that implicitly reset the retry count to 0? Will we get into cases where it retries until it fails, then the retry count is already == max_retry? * If a namenode is down when we try to resolve, we don't try again when it's time to fail over, do we? We should capture that in another bug For discussion, not necessarily to fix in this patch: * In FixedDealyWithFailover::ShouldRetry(), should we failover on any other errors other than timeout? Bad route to host? DNS failure? * In FixedDealyWithFailover::ShouldRetry(), we're always using a delay if retries < 3. This should be configurable. We can cover that in another bug > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-10441.HDFS-8707.010.patch, HDFS-10441.HDFS-8707.011.patch, > HDFS-10441.HDFS-8707.012.patch, HDFS-10441.HDFS-8707.013.patch, > HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376060#comment-15376060 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 42s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 23s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 26s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 42s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 47s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 54s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 49m 18s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12817832/HDFS-10441.HDFS-8707.013.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux d22cdda6748f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / d18e396 | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16053/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16053/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375865#comment-15375865 ] Xiaowei Zhu commented on HDFS-10441: The latest patch looks almost good to go with one small typo in rpc_engine.cc: {code} bool HANamenodeTracker::IsCurrentStandby_locked(const ::asio::ip::tcp::endpoint ) const { for(unsigned int i=0;ilibhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-10441.HDFS-8707.010.patch, HDFS-10441.HDFS-8707.011.patch, > HDFS-10441.HDFS-8707.012.patch, HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375380#comment-15375380 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 12s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 31s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 22s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 14s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 37s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 37s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 10s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 50m 30s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12817724/HDFS-10441.HDFS-8707.012.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux 25e8ac932b81 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / d18e396 | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16043/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16043/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373531#comment-15373531 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 47s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 26s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 24s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 35s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 36s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 41s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 48m 38s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12817481/HDFS-10441.HDFS-8707.011.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux 3e84fb7772e7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / d643d8c | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16030/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16030/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15371826#comment-15371826 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 22s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 45s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 24s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 21s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 42s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 38s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 30s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 59m 17s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12817262/HDFS-10441.HDFS-8707.010.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux 42d34237aefa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / d643d8c | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16019/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16019/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15359685#comment-15359685 ] James Clampffer commented on HDFS-10441: Thanks for the very detailed feedback Bob. You caught a lot of issues that slipped my mind + a few I didn't really think about. As far as bigger issues go (other points are also important to me, just want to give comments on these before the next patch): bq. Do we pass information in the data pointer for the NN failover event? If so, document it in events.h I think I accidentally left this out. Needs to be fixed, or I'll take it out of the header and add another jira for that. bq. rpc_connection.cc: HandleRpcResponse should push req back to the head of the queue; alternately, don't dequeue it if we got a standby exception. Need to take a better look at how to best do this. Will fix. bq. If HandleRpcResponse gets a kStandbyException, will CommsError be called twice (once in HandleRpcResponse and again in OnRecvComplete)? Yes, and I haven't figured out why yet, I should be able to track it down quickly now that I'm more familiar with the RPC retry code. It doesn't seem to break things but it sure makes reading the logs more confusing than they need to be. bq. rpc_engine.cc: let's use both namenodes if servers.size() >= 2 rather than just bailing out. I'm torn on this one. As far as I know you'd need to do a bit of hacking to get more than two nodes working in HA. It seems like it's best to cover the common case here. How about allowing more than 2 nodes but issuing warnings that say that your cluster probably isn't going to act as you'd expect based on the config file and that we discarded the extra nodes? bq. rpc_engine.h: IsCurrentActive/IsCurrentStandby are dangerous as designed: they're asking for race conditions as we acquire the lock, check, release the lock, then take action. Just before we take action, someone else could change the value You're totally right here. I didn't want to implement a mechanism to allow other things to lock the state of that structure for an arbitrary amount of time (seemed like it would be asking for a deadlock). I think the best fix here is to make them private to the NANamenodeTracker. bq. rpc_engine.cc: Remove RpcEngine::Start instead of deprecating it Yep, I left that in to see if someone could think of a use for a hook that got called once the IoService was known to be running. If you can't think of any I'll be happy to get rid of some dead code. bq. Don't forget to file bugs to handle more than two namenodes. Sure. I was thinking of holding off until that became widely supported but it won't hurt to plan ahead. As a general warning, I don't plan to have much internet access for the next week. Once I do I'll address the rest of the comments and get a new patch up that includes the rest of your feedback. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357832#comment-15357832 ] Bob Hansen commented on HDFS-10441: --- Small issues: all of these are small, but should probably be fixed before landing * Do we pass information in the data pointer for the NN failover event? If so, document it in events.h * Comment in retry_policy.h for FixedDelayWithFailover still references static values of 3 and 2 * We should include blocks of comments in retry_policy.h describing the behavior in human terms because the backoff behavior is less than obvious * status.cc: find the right value for kSnapshotException * filesystem.cc: nn_.Connect() call: get rid of commented-out code * rpc_connection.cc: HandleRpcResponse should push req back to the head of the queue; alternately, don't dequeue it if we got a standby exception. * If HandleRpcResponse gets a kStandbyException, will CommsError be called twice (once in HandleRpcResponse and again in OnRecvComplete)? * rpc_engine.cc: let's use both namenodes if servers.size() >= 2 rather than just bailing out. * rpc_engine.h: IsCurrentActive/IsCurrentStandby are dangerous as designed: they're asking for race conditions as we acquire the lock, check, release the lock, then take action. Just before we take action, someone else could change the value * rpc_engine.cc: Remove RpcEngine::Start instead of deprecating it. * Don't forget to file bugs to handle more than two namenodes. Minor issues: it would be nice to see these fixed, but aren't blockers: * status.h: is having both is_server_exception_ and exception_class_ redundant? * hdfs_configuration.c: We have a (faster) split function in uri.cc; let's refactor that into a Util method * HdfsConfiguration::LookupNameService: if the URI parsing failed, we should just ignore the URI as mal-formed, not bail out of the entire function. There may be a well-formed URI in a later value. * HdfsConfiguration: I'm a little uncomfortable using the URI parser to break apart host:port. If the user enters "foo:bar@baz", it will interpret that as a password and silently drop everything before the baz. Just using split(':') and converting the port to int if it exists is solid enough. * status.cc: I don't think the java exception name should go in the (user-visible) output message. A string describing the error ("Invalid Argument") would be nice, though. * filesystem.cc: why do we call InitRpc before checking if there's an io_service_? * rpc_engine.h: Are ha_persisted_info_ and ha_enabled_ redundant? > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357168#comment-15357168 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 36s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 48s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 39s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 24s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 14s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 49s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 45s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 53s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 64m 47s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12815458/HDFS-10441.HDFS-8707.009.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux d37e5df504ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / d643d8c | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15955/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15955/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353226#comment-15353226 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 21s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 59s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 45s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 50s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 51s{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 42s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 21s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 60m 20s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12814130/HDFS-10441.HDFS-8707.008.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux bf89e2cadf9c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / a903f78 | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15935/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15935/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15348701#comment-15348701 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 2m 47s{color} | {color:red} Docker failed to build yetus/hadoop:0cf5e66. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12813128/HDFS-10441.HDFS-8707.006.patch | | JIRA Issue | HDFS-10441 | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15907/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342686#comment-15342686 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 27s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 22s {color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 24s {color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 15s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 57s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 57s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 23s {color} | {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 35s {color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 54m 3s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_91 Failed CTEST tests | test_libhdfs_threaded_hdfspp_test_shim_static | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12812278/HDFS-10441.HDFS-8707.005.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux e495f055b6a4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / 71af408 | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/15855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_91.txt | | CTEST logs | https://builds.apache.org/job/PreCommit-HDFS-Build/15855/artifact/patchprocess/patch-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_91.ctest | | unit test logs | https://builds.apache.org/job/PreCommit-HDFS-Build/15855/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_91.txt | | JDK v1.7.0_101 Test Results |
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15330585#comment-15330585 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 35s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 26s {color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 26s {color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 38s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 39s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 7s {color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 12s {color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 16s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12810598/HDFS-10441.HDFS-8707.004.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux 8e41cacf59bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / 7c1d5df | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15767/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15767/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321651#comment-15321651 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 57s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 48s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 19s {color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 24s {color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 18s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 25s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 29s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 29s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 47s {color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 53s {color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 60m 19s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12809052/HDFS-10441.HDFS-8707.003.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux 0bcb38abd51f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / bfb2e27 | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15713/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15713/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321624#comment-15321624 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 42s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 23s {color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 22s {color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} HDFS-8707 passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 32s {color} | {color:green} the patch passed with JDK v1.8.0_91 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 38s {color} | {color:green} the patch passed with JDK v1.7.0_101 {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 38s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 14s {color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.8.0_91. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 6s {color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK v1.7.0_101. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 34s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0cf5e66 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12809048/HDFS-10441.HDFS-8707.002.patch | | JIRA Issue | HDFS-10441 | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux 0c410bf33592 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-8707 / bfb2e27 | | Default Java | 1.7.0_101 | | Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 | | JDK v1.7.0_101 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15711/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: hadoop-hdfs-project/hadoop-hdfs-native-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15711/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task >
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296249#comment-15296249 ] Bob Hansen commented on HDFS-10441: --- I realize that this was a "preview patch," and that there was a lot of cruft. I just wanted to give a nice checklist of "don't forget this before the Real Patch" lands. Checklists are awesome. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295860#comment-15295860 ] James Clampffer commented on HDFS-10441: Thanks for the really detailed review Bob! I'll have a new patch up tomorrow that should cover most of your comments, let me know if my responses sound in line with what you'd like. bq. Should Options::ha_namenodes_ be a mapto make it easier to look up? Yes, I just implemented this because the first round only dealt with 1 nameservice. The plan is to have a map > of nameservice IDs to namenodes in that service. Then add a getter that can look up NNs by service or default to whatever is specified by defaultFS. bq. Rather than have a server_exception_type, I would like to see the explicit error types as peers to our other error types. The fact that it came from a Java exception on the server side is an implementation detail of the communications. The kExceptionType should really be an "UnexpectedException" type, and all of the others should be mapped to semantically informative types (using an awesome lookup table, like you did) Agreed. This was partly because it let me make the assumption that if it happened on the server side it was probably a good time to fail over. Partly because I forgot how c++ enums worked.. I was concerned about having the int value of the enum entry collide with the std::errc codes but the compiler will deal with that. Would you be opposed to a method that would determine if an error likely originated on the server side? bq. Not super-critical, but it's a little bit cleaner and (possibly) faster to use strstr in HDFSConfiguration::SplitOnComma. Once you know the start and end, you can use std::string::string(const char *, int) to construct the string. Thanks for the reference, I hadn't used that. I'll fix this in an upcoming patch(little lower priority than the functional recommendations). bq. Instead of trying to manually parse the URI in ParsePartialURI, would it be easier to do a quick check for "://", and if it's not there, prepend DEFAULT_SCHEME (hdfs://), then pass it to the full-featured URI parser? Most likely. For now I'll go down that path. I've also filed HDFS-10411 because when the uri library fails to parse an incomplete url it will also jump on an ininitialized value according to valgrind. I'll post a more general fix to the parser in HDFS-10411. bq. By design the Configuration class is really a stupid container. If we can, all the logic should be in the HDFSConfigurationLoader. That sounds like a much better design than what I have, will do. bq. It seems that the nameservice lookups aren't actually URIs; they're just server:port addresses. Perphaps rather than storing a URI object, the HANamenodeInfo should just have a server and port field. The rest of the URI semantic space (scheme, user, password, path, fragment, etc.) really doesn't apply here, and would be an error if it appeared in the config file. I agree. I was hoping to avoid adding a new object to handle this case but with the trouble it's giving the parser it'd probably be worth it. bq. I would prefer "FormatURI" or "ToString" over "FmtURI" as a method name Yep, will do. FmtURI was a quick hack that I should have fixed up before posting. bq. I've noticed that we've grown used to "if (!stat.ok) { LOG_XXX(..., )". In trying to debug problems, I'm unhappy with me for not including the error message. As you touch any logging messages, << out the status.ToString() also I agree. The log messages on this last patch were a mess (thanks for the patience in reviewing). I'll include status::ToString in future messages; perhaps the logger should take a status object as an argument to enforce this. bq.As an extension of that: LOG_WARN(kRPC, << "Tried to connect to standby!"); What node was I trying to connect to? I'll clean this up. bq. EnqueueRequestsWithLock: I think elsewhere in the codebase we have used EnqueueRequests_locked; let's keep that standard Didn't see that, sounds like what I was looking for, I'll switch. bq. is_locked: we already have a lock_held method in util.h (that works with reentrant mutexes, too) Yea, I'll switch to that. I had the patch from HDFS-10409 applied and it made it into here. bq. I know we're going to add more-than-one HA NN later, but a good idiom is to keep them in a sorted vector, and have an index pointing to the current one. Then make next = (curr + 1) % size. Sounds good. That's only marginally more complicated than the std::swap but a lot more general. bq. Name lookup in ResoloveAllEndpoints needs to be async. This can be follow-up if necessary. I agree. I'd like to hold off on this in order to keep HA as simple as possible until it's well tested. bq. In general, having a recursive_mutex (HA NN Tracker) is a bad smell that we're
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294337#comment-15294337 ] Bob Hansen commented on HDFS-10441: --- Notes in chronological order; most of these are minor, nitpicky points. * Should Options::ha_namenodes_ be a mapto make it easier to look up? * Rather than have a server_exception_type, I would like to see the explicit error types as peers to our other error types. The fact that it came from a Java exception on the server side is an implementation detail of the communications. The kExceptionType should really be an "UnexpectedException" type, and all of the others should be mapped to semantically informative types (using an awesome lookup table, like you did) * Not super-critical, but it's a little bit cleaner and (possibly) faster to use strstr in HDFSConfiguration::SplitOnComma. Once you know the start and end, you can use std::string::string(const char *, int) to construct the string. * Instead of trying to manually parse the URI in ParsePartialURI, would it be easier to do a quick check for "://", and if it's not there, prepend DEFAULT_SCHEME (hdfs://), then pass it to the full-featured URI parser? * By design the Configuration class is really a stupid container. If we can, all the logic should be in the HDFSConfigurationLoader. * It seems that the nameservice lookups aren't actually URIs; they're just server:port addresses. Perphaps rather than storing a URI object, the HANamenodeInfo should just have a server and port field. The rest of the URI semantic space (scheme, user, password, path, fragment, etc.) really doesn't apply here, and would be an error if it appeared in the config file. * I would prefer "FormatURI" or "ToString" over "FmtURI" as a method name * I've noticed that we've grown used to "if (!stat.ok) { LOG_XXX(..., )". In trying to debug problems, I'm unhappy with me for not including the error message. As you touch any logging messages, << out the status.ToString() also * As an extension of that: LOG_WARN(kRPC, << "Tried to connect to standby!"); What node was I trying to connect to? * EnqueueRequestsWithLock: I think elsewhere in the codebase we have used EnqueueRequests_locked; let's keep that standard * is_locked: we already have a lock_held method in util.h (that works with reentrant mutexes, too) * I know we're going to add more-than-one HA NN later, but a good idiom is to keep them in a sorted vector, and have an index pointing to the current one. Then make next = (curr + 1) % size. * Name lookup in ResoloveAllEndpoints needs to be async. This can be follow-up if necessary. * In general, having a recursive_mutex (HA NN Tracker) is a bad smell that we're not thinking about concurrency tightly enough. I'm not sure that's the case here, but let's think about it. * Make sure to clean up TODOs and anything referring to internal server names * Unset port should be a WARNING, if we're going to default and move on * In RpcEngine::Start, if resolution fails and we clear out ha_persisted_info_, what will happen when we run? * Better logging than "" and "Caught this early" * In RpcEngine::RpcCommsError, we should try for symmetry in retry count and failure count. Let's pull the IncrementRetryCount out of being inline, and just use both getters in the call to ShouldRetry. * You have a comment for eventually failing out messages if we're bouncing between standby nodes, but it looks like we only actually check if it's not a kStandbyException. If I'm mis-reading, let me know. * We don't need the first line here, do we? + last_endpoints_.clear(); + last_endpoints_ = new_active_nn_info.endpoints; * In RpcConnection::HandleRpcResponse, I think we should be checking status and erroring out before calling RemoveFromRunningQueue. We should look at this closely, and then we could get rid of the EnqueueReqeustsWithLock method. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294321#comment-15294321 ] Hadoop QA commented on HDFS-10441: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color} | {color:red} HDFS-10441 does not apply to HDFS-8707. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12805246/HDFS-10441.HDFS-8707.000.patch | | JIRA Issue | HDFS-10441 | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15516/console | | Powered by | Apache Yetus 0.2.0 http://yetus.apache.org | This message was automatically generated. > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer > Attachments: HDFS-10441.HDFS-8707.000.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org