[jira] [Commented] (HDFS-6329) WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[ https://issues.apache.org/jira/browse/HDFS-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993049#comment-13993049 ] Kihwal Lee commented on HDFS-6329: -- Ouch. Now you know what I use as a debug marker in logs. I will take it out. bq. Maybe I missed it, but a test cast to simulate our HA IP-failover would be invaluable to preventing another regression. That's very tricky with the current state of MiniDFSCluster and other broken checks in HA. HDFS-6360 will fix MiniDFSCluster and HDFS-6334 will fix HA checks. Perhap we can add more meaningful tests after that. > WebHdfs does not work if HA is enabled on NN but logical URI is not > configured. > --- > > Key: HDFS-6329 > URL: https://issues.apache.org/jira/browse/HDFS-6329 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Fix For: 3.0.0, 2.4.1 > > Attachments: HDFS-6329.patch, HDFS-6329.patch, HDFS-6329.v2.patch, > HDFS-6329.v3.patch, HDFS-6329.v4.patch, HDFS-6329.v5.patch > > > After HDFS-6100, namenode unconditionally puts the logical name (name service > id) as the token service when redirecting webhdfs requests to datanodes, if > it detects HA. > For HA configurations with no client-side failover proxy provider (e.g. IP > failover), webhdfs does not work since the clients do not use logical name. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6329) WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[ https://issues.apache.org/jira/browse/HDFS-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992932#comment-13992932 ] Kihwal Lee commented on HDFS-6329: -- bq. We should really sort things out and make test cases do the right thing instead of relying on side-effects of shared conf \[ in MiniDFSCluster\]. But that's for another day and I am simply making it behave like the old way for now. I saw unit test failures in other JIRAs due to this issue, so I filed HDFS-6360. > WebHdfs does not work if HA is enabled on NN but logical URI is not > configured. > --- > > Key: HDFS-6329 > URL: https://issues.apache.org/jira/browse/HDFS-6329 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6329.patch, HDFS-6329.patch, HDFS-6329.v2.patch, > HDFS-6329.v3.patch, HDFS-6329.v4.patch > > > After HDFS-6100, namenode unconditionally puts the logical name (name service > id) as the token service when redirecting webhdfs requests to datanodes, if > it detects HA. > For HA configurations with no client-side failover proxy provider (e.g. IP > failover), webhdfs does not work since the clients do not use logical name. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6329) WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[ https://issues.apache.org/jira/browse/HDFS-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994349#comment-13994349 ] Hudson commented on HDFS-6329: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #560 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/560/]) HDFS-6329. WebHdfs does not work if HA is enabled on NN but logical URI is not configured. Contributed by Kihwal Lee. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593470) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java > WebHdfs does not work if HA is enabled on NN but logical URI is not > configured. > --- > > Key: HDFS-6329 > URL: https://issues.apache.org/jira/browse/HDFS-6329 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Fix For: 3.0.0, 2.4.1 > > Attachments: HDFS-6329.patch, HDFS-6329.patch, HDFS-6329.v2.patch, > HDFS-6329.v3.patch, HDFS-6329.v4.patch, HDFS-6329.v5.patch > > > After HDFS-6100, namenode unconditionally puts the logical name (name service > id) as the token service when redirecting webhdfs requests to datanodes, if > it detects HA. > For HA configurations with no client-side failover proxy provider (e.g. IP > failover), webhdfs does not work since the clients do not use logical name. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6329) WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[ https://issues.apache.org/jira/browse/HDFS-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994393#comment-13994393 ] Hudson commented on HDFS-6329: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1752 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1752/]) HDFS-6329. WebHdfs does not work if HA is enabled on NN but logical URI is not configured. Contributed by Kihwal Lee. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593470) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java > WebHdfs does not work if HA is enabled on NN but logical URI is not > configured. > --- > > Key: HDFS-6329 > URL: https://issues.apache.org/jira/browse/HDFS-6329 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Fix For: 3.0.0, 2.4.1 > > Attachments: HDFS-6329.patch, HDFS-6329.patch, HDFS-6329.v2.patch, > HDFS-6329.v3.patch, HDFS-6329.v4.patch, HDFS-6329.v5.patch > > > After HDFS-6100, namenode unconditionally puts the logical name (name service > id) as the token service when redirecting webhdfs requests to datanodes, if > it detects HA. > For HA configurations with no client-side failover proxy provider (e.g. IP > failover), webhdfs does not work since the clients do not use logical name. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6329) WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[ https://issues.apache.org/jira/browse/HDFS-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994379#comment-13994379 ] Hudson commented on HDFS-6329: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1778 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1778/]) HDFS-6329. WebHdfs does not work if HA is enabled on NN but logical URI is not configured. Contributed by Kihwal Lee. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593470) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java > WebHdfs does not work if HA is enabled on NN but logical URI is not > configured. > --- > > Key: HDFS-6329 > URL: https://issues.apache.org/jira/browse/HDFS-6329 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Fix For: 3.0.0, 2.4.1 > > Attachments: HDFS-6329.patch, HDFS-6329.patch, HDFS-6329.v2.patch, > HDFS-6329.v3.patch, HDFS-6329.v4.patch, HDFS-6329.v5.patch > > > After HDFS-6100, namenode unconditionally puts the logical name (name service > id) as the token service when redirecting webhdfs requests to datanodes, if > it detects HA. > For HA configurations with no client-side failover proxy provider (e.g. IP > failover), webhdfs does not work since the clients do not use logical name. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6329) WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[
https://issues.apache.org/jira/browse/HDFS-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993026#comment-13993026
]
Daryn Sharp commented on HDFS-6329:
---
You may want to remove this line {{LOG.info("KLKL uri = " + uri);}}. :)
Maybe I missed it, but a test cast to simulate our HA IP-failover would be
invaluable to preventing another regression.
> WebHdfs does not work if HA is enabled on NN but logical URI is not
> configured.
> ---
>
> Key: HDFS-6329
> URL: https://issues.apache.org/jira/browse/HDFS-6329
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Fix For: 3.0.0, 2.4.1
>
> Attachments: HDFS-6329.patch, HDFS-6329.patch, HDFS-6329.v2.patch,
> HDFS-6329.v3.patch, HDFS-6329.v4.patch, HDFS-6329.v5.patch
>
>
> After HDFS-6100, namenode unconditionally puts the logical name (name service
> id) as the token service when redirecting webhdfs requests to datanodes, if
> it detects HA.
> For HA configurations with no client-side failover proxy provider (e.g. IP
> failover), webhdfs does not work since the clients do not use logical name.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6329) WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[
https://issues.apache.org/jira/browse/HDFS-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13991416#comment-13991416
]
Hadoop QA commented on HDFS-6329:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12643640/HDFS-6329.v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
org.apache.hadoop.hdfs.server.namenode.TestNameNodeRecovery
org.apache.hadoop.hdfs.server.namenode.TestEditLogAutoroll
org.apache.hadoop.hdfs.TestDistributedFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6842//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6842//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6842//console
This message is automatically generated.
> WebHdfs does not work if HA is enabled on NN but logical URI is not
> configured.
> ---
>
> Key: HDFS-6329
> URL: https://issues.apache.org/jira/browse/HDFS-6329
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Attachments: HDFS-6329.patch, HDFS-6329.patch, HDFS-6329.v2.patch,
> HDFS-6329.v3.patch
>
>
> After HDFS-6100, namenode unconditionally puts the logical name (name service
> id) as the token service when redirecting webhdfs requests to datanodes, if
> it detects HA.
> For HA configurations with no client-side failover proxy provider (e.g. IP
> failover), webhdfs does not work since the clients do not use logical name.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6329) WebHdfs does not work if HA is enabled on NN but logical URI is not configured.
[
https://issues.apache.org/jira/browse/HDFS-6329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988785#comment-13988785
]
Hadoop QA commented on HDFS-6329:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12643177/HDFS-6329.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.shell.TestHdfsTextCommand
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
org.apache.hadoop.hdfs.server.namenode.TestAclConfigFlag
org.apache.hadoop.hdfs.TestClose
org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
org.apache.hadoop.hdfs.TestFSInputChecker
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.TestDataTransferProtocol
org.apache.hadoop.hdfs.server.namenode.TestHDFSConcat
org.apache.hadoop.hdfs.server.namenode.TestNamenodeCapacityReport
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
org.apache.hadoop.hdfs.server.namenode.TestFSNamesystemMBean
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters
org.apache.hadoop.tools.TestJMXGet
org.apache.hadoop.hdfs.TestDFSShell
org.apache.hadoop.hdfs.server.namenode.TestFSImage
org.apache.hadoop.hdfs.security.TestDelegationToken
org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
org.apache.hadoop.fs.permission.TestStickyBit
org.apache.hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional
org.apache.hadoop.hdfs.TestFileConcurrentReader
org.apache.hadoop.hdfs.server.datanode.TestCachingStrategy
org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
org.apache.hadoop.fs.TestFcHdfsCreateMkdir
org.apache.hadoop.hdfs.TestCrcCorruption
org.apache.hadoop.hdfs.TestAppendDifferentChecksum
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration
org.apache.hadoop.hdfs.server.namenode.TestParallelImageWrite
org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
org.apache.hadoop.hdfs.server.namenode.TestSequentialBlockId
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap
org.apache.hadoop.hdfs.tools.TestGetGroups
org.apache.hadoop.hdfs.TestDFSPermission
org.apache.hadoop.hdfs.server.namenode.snapshot.TestDisallowModifyROSnapshot
org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
org.apache.hadoop.hdfs.TestListFilesInFileContext
org.apache.hadoop.hdfs.server.namenode.TestNameNodeResourceChecker
org.apache.hadoop.fs.viewfs.TestViewFsDefaultValue
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot
org.apache.hadoop.hdfs.TestParallelShortCircuitReadNoChecksum
org.apache.hadoop.hdfs.TestDatanodeReport
org.apache.hadoop.hdfs.TestDFSRemove
org.apache.hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage
org
