[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771204#comment-13771204
]
Jing Zhao commented on HDFS-5122:
-
+1 for the new patch. I will commit it soon.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13771174#comment-13771174
]
Hadoop QA commented on HDFS-5122:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603868/HDFS-5122.004.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4996//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4996//console
This message is automatically generated.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.004.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769828#comment-13769828
]
Suresh Srinivas commented on HDFS-5122:
---
[~wheat9], nice job.
Comments:
# WebHdfsFileSystem should not refer to DFSClient. I recommend just
initializing such variables with in WebHdfsFileSystem.
# Please add unit tests for newly added DFSUtil methods.
# nit: real name node to real namenode
# While changing this code, we should add some documentation to the retry
logic. Especially how getResponse works
# Other changes unrelated to the code
#* DFSUtil - remove unused imports HashMap, TimeUnit,
AlreadyBeingCreatedException, RetryPolicies, RetryPolicy, RetryProxy.
#* WebHdfsFileSystem - remove unused imports AuthorizationException,
InvalidToken
#* WebHdfsFileSystem - use constant SCHEM in getScheme() instead of hardcoded
constant.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770216#comment-13770216
]
Hadoop QA commented on HDFS-5122:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603721/HDFS-5122.003.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4988//console
This message is automatically generated.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13770320#comment-13770320
]
Hadoop QA commented on HDFS-5122:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12603721/HDFS-5122.003.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4990//console
This message is automatically generated.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.002.patch,
HDFS-5122.003.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13766128#comment-13766128
]
Jing Zhao commented on HDFS-5122:
-
The new patch looks pretty good to me. Some further comments:
1.
{code}
+} else {
+ Conf config = new Conf(conf);
+ this.retryPolicy = RetryPolicies
+ .failoverOnNetworkException(RetryPolicies.TRY_ONCE_THEN_FAIL,
+ config.maxFailoverAttempts, config.failoverSleepBaseMillis,
+ config.failoverSleepMaxMillis);
{code}
Here we use the same configuration properties for DFSClient. Not sure if we
need to create some new configuration properties for
WebHDFSFileSystem?
2.
{code}
+ private synchronized void resetStateToFailOver() {
+currentNNAddrIndex = (currentNNAddrIndex + 1) % nnAddrs.length;
+delegationToken = null;
+hasInitedToken = false;
+ }
{code}
Can we cache the delegationToken for each NN here, and re-fetch it only when
necessary? (Possibly this can be addressed in a separate jira)
3. In the current patch the class HDFSUrlResolver acts as a helper class. So we
can move the static method to DFSUtil and remove the class.
4.
{code}
+ } else if (a.action ==
RetryPolicy.RetryAction.RetryDecision.FAILOVER_AND_RETRY) {
+resetStateToFailOver();
+Thread.sleep(a.delayMillis);
+return;
{code}
We can add a log here. Besides, currently the isIdempotentOrAtMostOnce
parameter for RetryPolicy#shouldRetry is directly set to true. I went through
all the WebHDFSFileSystem methods and looks like it will be fine. If there is
other issue we can also address it later.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13764707#comment-13764707
]
Alejandro Abdelnur commented on HDFS-5122:
--
I would take a slightly different approach for this:
The WebHdfsFileSystem#getHttpUrlConnection() method is used for every FS call.
In this method, if the URI is for the NN (metadata operation, data transfers
will be to a DN) do the following:
1* the standby NM should have a redirector HTTP servlet that bounces all HTTP
calls from the standby to the active.
2* Use the same utility classes DistributedFileSystem class uses to determine
if hostname in the URI is a logical name or not.
3* If the hostname is no a logical name, do current logic.
4* If the hostname is a logical name, resolve to any of the NN hosts, do a
cheap FS call using the chosen hostname.
4.1**If it works cache the chosen hostname and use it for all subsequent FS
operations while successful.
4.2** If the call returns a redirect (automatic redirects are disabled) means
you hit the standby, select the other hostname and use it for all subsequent FS
operations while successful.
4.3** if the call returns a cannot connect or error means you hit a NN that is
'dead', fallback to the other NN hostname and use it for all subsequent FS
operations while successful.
5* When a subsequent URL call fails do #4.3
6* Make sure you have logic to avoid infinite loop of bouncing between NNs in
case both are dead.
7*The WebHDFS delegation token service should use similar logic like the DFS
delegation token to convert from logical name to hostname the service in the
token.
NOTE: I'm not familiar on WebHDFSFileSystem current retry logic, and some of
this could be already take can of it.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13764788#comment-13764788
]
Jing Zhao commented on HDFS-5122:
-
bq. 1* the standby NM should have a redirector HTTP servlet that bounces all
HTTP calls from the standby to the active.
Maybe we do not need to change the NN side here. A WebHdfsFileSystem call can
get StandbyException if it hits the SBN, and we can use it for
WebHdfsFileSystem side's failover. This is the same logic with current
DFSClient's failover and retry mechanism.
bq. 4* If the hostname is a logical name, resolve to any of the NN hosts, do a
cheap FS call using the chosen hostname.
bq. 4.1**If it works cache the chosen hostname and use it for all subsequent FS
operations while successful.
After the cheap call, we cannot guarantee that the subsequence calls can
succeed since the NN failover can happen in between. So I think we can skip the
cheap call here.
The current WebHDFSFileSystem retry logic does not consider the NN failover. So
I think we only need to add the failover (between the 2 NN URL) and retry
logic, and use FailoverOnNetworkExceptionRetry as the retry policy in
WebHDFSFileSystem for HA setup. Looks like this is what Haohui is doing in his
current patch.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13764801#comment-13764801
]
Haohui Mai commented on HDFS-5122:
--
Hi Alejandro,
I believe that this patch realizes your intuitions from 3~6, mostly reusing the
current retry logic of WebHDFS.
For (1), can you please elaborate your ideas on why introducing an additional
redirector on the server side? Right now the client will get a StandbyException
and simply retry.
For (2), I appreciate if you could kindly give me a pointer to the actual
method.
For (7), this patch forgets about the delegation token when the client connects
to a different name node. Based on my understanding it should be a simple
approach to get things working. I'm wondering whether WebHDFS needs a more
sophisticated approach here. Can you elaborate how the DFS client handle this
case?
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13764923#comment-13764923
]
Hadoop QA commented on HDFS-5122:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602663/HDFS-5122.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4957//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4957//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4957//console
This message is automatically generated.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13765110#comment-13765110
]
Hadoop QA commented on HDFS-5122:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602711/HDFS-5122.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4958//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4958//console
This message is automatically generated.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.001.patch, HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763585#comment-13763585
]
Jing Zhao commented on HDFS-5122:
-
The patch looks very good to me. Some minor comments:
1. Need to check the indentation of NameNodeProxies#isHAConfiguration.
2. Looks like we do not need to catch the IOException within
NameNodeProxies#isHAConfiguration. Otherwise the IOException thrown by
getFailoverProxyProviderClass may be swallowed.
3. In WbeHdfsFileSystem#initialize,
{code}
Preconditions.checkNotNull(namenodes);
Preconditions.checkArgument(namenodes.size() 0,
Does not contain HA HTTP server);
{code}
may be combined in a single Preconditions.checkState call.
4. In TestWebHDFSForHA.java, LOG has not been used, and it may be better to
declare LOGICAL_NAME outside since in the future we can reuse it.
5. bq. Putting the work of automatically switching to ...
Sounds good. We can add support for client side's failover to
WebHdfsFilesystem in a separate jira.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763665#comment-13763665
]
Philip Zeyliger commented on HDFS-5122:
---
Haohui,
Your patch doesn't seem to handle the possibility that the first namenode isn't
the currently active one. So it's not just not handling failover during the
lifetime of the client, but it's not handling a case where the configuration
happens to have the currently active namenode first. It seems like the code
ought to at least try to access (with a timeout) the namenode to see if it's
active, no?
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763708#comment-13763708
]
Hadoop QA commented on HDFS-5122:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602425/HDFS-5122.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4949//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4949//console
This message is automatically generated.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763727#comment-13763727
]
Haohui Mai commented on HDFS-5122:
--
I've created a new JIRA to track the status of the fail-over mechanism in
WebHDFS:
https://issues.apache.org/jira/browse/HDFS-5181
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763756#comment-13763756
]
Andrew Wang commented on HDFS-5122:
---
I agree with Phil, this patch isn't very useful as-is if it can't handle the
first namenode being down. Since we're going to need this in very short order,
can we just roll-up HDFS-5181 into this JIRA?
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763771#comment-13763771
]
Hadoop QA commented on HDFS-5122:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602439/HDFS-5122.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4951//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/4951//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4951//console
This message is automatically generated.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763721#comment-13763721
]
Jing Zhao commented on HDFS-5122:
-
bq. to handle the possibility that the first namenode isn't the currently
active one
I think that's what Haohui's comment Putting the work of automatically
switching to means here. So we may have some bad scenarios such as 1) the
first listed namenode is not active right now, and 2) the NN failover is
happening thus the previous active NN becomes standby, and 3) the active NN
keeps changing in a short period of time.. Thus what we need here is the same
logic with RetryInvocationHandler.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-5122) WebHDFS should support logical service names in URIs
[
https://issues.apache.org/jira/browse/HDFS-5122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13763802#comment-13763802
]
Hadoop QA commented on HDFS-5122:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602449/HDFS-5122.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4952//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4952//console
This message is automatically generated.
WebHDFS should support logical service names in URIs
Key: HDFS-5122
URL: https://issues.apache.org/jira/browse/HDFS-5122
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.1.0-beta
Reporter: Arpit Gupta
Assignee: Haohui Mai
Attachments: HDFS-5122.patch
For example if the dfs.nameservices is set to arpit
{code}
hdfs dfs -ls webhdfs://arpit:50070/tmp
or
hdfs dfs -ls webhdfs://arpit/tmp
{code}
does not work
You have to provide the exact active namenode hostname. On an HA cluster
using dfs client one should not need to provide the active nn hostname
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
