[jira] [Commented] (HDFS-5723) Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction
[ https://issues.apache.org/jira/browse/HDFS-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084375#comment-14084375 ] Hudson commented on HDFS-5723: -- FAILURE: Integrated in Hadoop-trunk-Commit #6005 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6005/]) HDFS-5723. Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction. Contributed by Vinayakumar B. (vinayakumarb: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615491) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRetryCacheWithHA.java > Append failed FINALIZED replica should not be accepted as valid when that > block is underconstruction > > > Key: HDFS-5723 > URL: https://issues.apache.org/jira/browse/HDFS-5723 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Fix For: 2.6.0 > > Attachments: HDFS-5723.patch, HDFS-5723.patch, HDFS-5723.patch > > > Scenario: > 1. 3 node cluster with > dfs.client.block.write.replace-datanode-on-failure.enable set to false. > 2. One file is written with 3 replicas, blk_id_gs1 > 3. One of the datanode DN1 is down. > 4. File was opened with append and some more data is added to the file and > synced. (to only 2 live nodes DN2 and DN3)-- blk_id_gs2 > 5. Now DN1 restarted > 6. In this block report, DN1 reported FINALIZED block blk_id_gs1, this should > be marked corrupted. > but since NN having appended block state as UnderConstruction, at this time > its not detecting this block as corrupt and adding to valid block locations. > As long as the namenode is alive, this datanode also will be considered as > valid replica and read/append will fail in that datanode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6247) Avoid timeouts for replaceBlock() call by sending intermediate responses to Balancer
[ https://issues.apache.org/jira/browse/HDFS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084372#comment-14084372 ] Vinayakumar B commented on HDFS-6247: - Hi [~umamaheswararao], Can you please take a look at the patch if possible.. ? thanks > Avoid timeouts for replaceBlock() call by sending intermediate responses to > Balancer > > > Key: HDFS-6247 > URL: https://issues.apache.org/jira/browse/HDFS-6247 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer, datanode >Affects Versions: 2.4.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-6247.patch, HDFS-6247.patch, HDFS-6247.patch, > HDFS-6247.patch, HDFS-6247.patch, HDFS-6247.patch > > > Currently there is no response sent from target Datanode to Balancer for the > replaceBlock() calls. > Since the Block movement for balancing is throttled, complete block movement > will take time and this could result in timeout at Balancer, which will be > trying to read the status message. > > To Avoid this during replaceBlock() call in in progress Datanode can send > IN_PROGRESS status messages to Balancer to avoid timeouts and treat > BlockMovement as failed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5723) Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction
[ https://issues.apache.org/jira/browse/HDFS-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084371#comment-14084371 ] Vinayakumar B commented on HDFS-5723: - Committed to trunk and branch-2. Thanks [~umamaheswararao], [~jingzhao] and [~stanley_shi] > Append failed FINALIZED replica should not be accepted as valid when that > block is underconstruction > > > Key: HDFS-5723 > URL: https://issues.apache.org/jira/browse/HDFS-5723 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Fix For: 2.6.0 > > Attachments: HDFS-5723.patch, HDFS-5723.patch, HDFS-5723.patch > > > Scenario: > 1. 3 node cluster with > dfs.client.block.write.replace-datanode-on-failure.enable set to false. > 2. One file is written with 3 replicas, blk_id_gs1 > 3. One of the datanode DN1 is down. > 4. File was opened with append and some more data is added to the file and > synced. (to only 2 live nodes DN2 and DN3)-- blk_id_gs2 > 5. Now DN1 restarted > 6. In this block report, DN1 reported FINALIZED block blk_id_gs1, this should > be marked corrupted. > but since NN having appended block state as UnderConstruction, at this time > its not detecting this block as corrupt and adding to valid block locations. > As long as the namenode is alive, this datanode also will be considered as > valid replica and read/append will fail in that datanode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5723) Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction
[ https://issues.apache.org/jira/browse/HDFS-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-5723: Resolution: Fixed Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Append failed FINALIZED replica should not be accepted as valid when that > block is underconstruction > > > Key: HDFS-5723 > URL: https://issues.apache.org/jira/browse/HDFS-5723 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Fix For: 2.6.0 > > Attachments: HDFS-5723.patch, HDFS-5723.patch, HDFS-5723.patch > > > Scenario: > 1. 3 node cluster with > dfs.client.block.write.replace-datanode-on-failure.enable set to false. > 2. One file is written with 3 replicas, blk_id_gs1 > 3. One of the datanode DN1 is down. > 4. File was opened with append and some more data is added to the file and > synced. (to only 2 live nodes DN2 and DN3)-- blk_id_gs2 > 5. Now DN1 restarted > 6. In this block report, DN1 reported FINALIZED block blk_id_gs1, this should > be marked corrupted. > but since NN having appended block state as UnderConstruction, at this time > its not detecting this block as corrupt and adding to valid block locations. > As long as the namenode is alive, this datanode also will be considered as > valid replica and read/append will fail in that datanode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5723) Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction
[ https://issues.apache.org/jira/browse/HDFS-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084364#comment-14084364 ] Vinayakumar B commented on HDFS-5723: - Thanks Uma for the review. I will commit this soon > Append failed FINALIZED replica should not be accepted as valid when that > block is underconstruction > > > Key: HDFS-5723 > URL: https://issues.apache.org/jira/browse/HDFS-5723 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-5723.patch, HDFS-5723.patch, HDFS-5723.patch > > > Scenario: > 1. 3 node cluster with > dfs.client.block.write.replace-datanode-on-failure.enable set to false. > 2. One file is written with 3 replicas, blk_id_gs1 > 3. One of the datanode DN1 is down. > 4. File was opened with append and some more data is added to the file and > synced. (to only 2 live nodes DN2 and DN3)-- blk_id_gs2 > 5. Now DN1 restarted > 6. In this block report, DN1 reported FINALIZED block blk_id_gs1, this should > be marked corrupted. > but since NN having appended block state as UnderConstruction, at this time > its not detecting this block as corrupt and adding to valid block locations. > As long as the namenode is alive, this datanode also will be considered as > valid replica and read/append will fail in that datanode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6814) Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean
[
https://issues.apache.org/jira/browse/HDFS-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084349#comment-14084349
]
Vinayakumar B commented on HDFS-6814:
-
+1, lgtm
> Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as
> boolean
> -
>
> Key: HDFS-6814
> URL: https://issues.apache.org/jira/browse/HDFS-6814
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-6814.patch
>
>
> {code}
>
> dfs.namenode.list.encryption.zones.num.responses
> false
> When listing encryption zones, the maximum number of zones
> that will be returned in a batch. Fetching the list incrementally in
> batches improves namenode performance.
>
>
> {code}
> default value should be 100. Should be same as {code}public static final int
> DFS_NAMENODE_LIST_ENCRYPTION_ZONES_NUM_RESPONSES_DEFAULT = 100;{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5185) DN fails to startup if one of the data dir is full
[
https://issues.apache.org/jira/browse/HDFS-5185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084293#comment-14084293
]
Uma Maheswara Rao G commented on HDFS-5185:
---
+1 latest patch looks good to me. Pending jenkins.
> DN fails to startup if one of the data dir is full
> --
>
> Key: HDFS-5185
> URL: https://issues.apache.org/jira/browse/HDFS-5185
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-5185-002.patch, HDFS-5185-003.patch, HDFS-5185.patch
>
>
> DataNode fails to startup if one of the data dirs configured is out of space.
> fails with following exception
> {noformat}2013-09-11 17:48:43,680 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> block pool Block pool (storage id
> DS-308316523-xx.xx.xx.xx-64015-1378896293604) service to /nn1:65110
> java.io.IOException: Mkdirs failed to create
> /opt/nish/data/current/BP-123456-1234567/tmp
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.(BlockPoolSlice.java:105)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addBlockPool(FsVolumeImpl.java:216)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.addBlockPool(FsVolumeList.java:155)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.addBlockPool(FsDatasetImpl.java:1593)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:834)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:311)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:217)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:660)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> It should continue to start-up with other data dirs available.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-5185) DN fails to startup if one of the data dir is full
[
https://issues.apache.org/jira/browse/HDFS-5185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-5185:
Attachment: HDFS-5185-003.patch
As suggested by [~umamaheswararao] offline,
Changed existing {{checkDiskError()}} to {{checkDiskErrorAsync()}}
and named new sync method to {{checkDiskError()}}.
> DN fails to startup if one of the data dir is full
> --
>
> Key: HDFS-5185
> URL: https://issues.apache.org/jira/browse/HDFS-5185
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-5185-002.patch, HDFS-5185-003.patch, HDFS-5185.patch
>
>
> DataNode fails to startup if one of the data dirs configured is out of space.
> fails with following exception
> {noformat}2013-09-11 17:48:43,680 FATAL
> org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
> block pool Block pool (storage id
> DS-308316523-xx.xx.xx.xx-64015-1378896293604) service to /nn1:65110
> java.io.IOException: Mkdirs failed to create
> /opt/nish/data/current/BP-123456-1234567/tmp
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.(BlockPoolSlice.java:105)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addBlockPool(FsVolumeImpl.java:216)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.addBlockPool(FsVolumeList.java:155)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.addBlockPool(FsDatasetImpl.java:1593)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:834)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:311)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:217)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:660)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
> It should continue to start-up with other data dirs available.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6663) Admin command to track file and locations from block id
[ https://issues.apache.org/jira/browse/HDFS-6663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084242#comment-14084242 ] Chen He commented on HDFS-6663: --- done with the preliminary code change, right now is working on unit test. > Admin command to track file and locations from block id > --- > > Key: HDFS-6663 > URL: https://issues.apache.org/jira/browse/HDFS-6663 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Kihwal Lee >Assignee: Chen He > > A dfsadmin command that allows finding out the file and the locations given a > block number will be very useful in debugging production issues. It may be > possible to add this feature to Fsck, instead of creating a new command. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[ https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HDFS-6790: -- Status: Patch Available (was: Open) > DFSUtil Should Use configuration.getPassword for SSL passwords > -- > > Key: HDFS-6790 > URL: https://issues.apache.org/jira/browse/HDFS-6790 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Larry McCay > Attachments: HDFS-6790.patch, HDFS-6790.patch > > > As part of HADOOP-10904, DFSUtil should be changed to leverage the new method > on Configuration for acquiring known passwords for SSL. The getPassword > method will leverage the credential provider API and/or fallback to the clear > text value stored in ssl-server.xml. > This will provide an alternative to clear text passwords on disk while > maintaining backward compatibility for this behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[ https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HDFS-6790: -- Status: Open (was: Patch Available) > DFSUtil Should Use configuration.getPassword for SSL passwords > -- > > Key: HDFS-6790 > URL: https://issues.apache.org/jira/browse/HDFS-6790 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Larry McCay > Attachments: HDFS-6790.patch, HDFS-6790.patch > > > As part of HADOOP-10904, DFSUtil should be changed to leverage the new method > on Configuration for acquiring known passwords for SSL. The getPassword > method will leverage the credential provider API and/or fallback to the clear > text value stored in ssl-server.xml. > This will provide an alternative to clear text passwords on disk while > maintaining backward compatibility for this behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[ https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084196#comment-14084196 ] Larry McCay commented on HDFS-6790: --- Test failure is unrelated to the patch. > DFSUtil Should Use configuration.getPassword for SSL passwords > -- > > Key: HDFS-6790 > URL: https://issues.apache.org/jira/browse/HDFS-6790 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Larry McCay > Attachments: HDFS-6790.patch, HDFS-6790.patch > > > As part of HADOOP-10904, DFSUtil should be changed to leverage the new method > on Configuration for acquiring known passwords for SSL. The getPassword > method will leverage the credential provider API and/or fallback to the clear > text value stored in ssl-server.xml. > This will provide an alternative to clear text passwords on disk while > maintaining backward compatibility for this behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[
https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084172#comment-14084172
]
Hadoop QA commented on HDFS-6790:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659585/HDFS-6790.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7551//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7551//console
This message is automatically generated.
> DFSUtil Should Use configuration.getPassword for SSL passwords
> --
>
> Key: HDFS-6790
> URL: https://issues.apache.org/jira/browse/HDFS-6790
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Larry McCay
> Attachments: HDFS-6790.patch, HDFS-6790.patch
>
>
> As part of HADOOP-10904, DFSUtil should be changed to leverage the new method
> on Configuration for acquiring known passwords for SSL. The getPassword
> method will leverage the credential provider API and/or fallback to the clear
> text value stored in ssl-server.xml.
> This will provide an alternative to clear text passwords on disk while
> maintaining backward compatibility for this behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6694) TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently with various symptoms
[ https://issues.apache.org/jira/browse/HDFS-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084148#comment-14084148 ] Yongjun Zhang commented on HDFS-6694: - Created INFRA-8147: check on open file limit on jenkins test slaves. > TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently > with various symptoms > > > Key: HDFS-6694 > URL: https://issues.apache.org/jira/browse/HDFS-6694 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Yongjun Zhang > Attachments: > org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover-output.txt, > org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.txt > > > TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently > with various symptoms. Typical failures are described in first comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6451) NFS should not return NFS3ERR_IO for AccessControlException
[
https://issues.apache.org/jira/browse/HDFS-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084140#comment-14084140
]
Hadoop QA commented on HDFS-6451:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659584/HDFS-6451.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7550//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7550//console
This message is automatically generated.
> NFS should not return NFS3ERR_IO for AccessControlException
>
>
> Key: HDFS-6451
> URL: https://issues.apache.org/jira/browse/HDFS-6451
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Reporter: Brandon Li
> Attachments: HDFS-6451.002.patch, HDFS-6451.003.patch, HDFS-6451.patch
>
>
> As [~jingzhao] pointed out in HDFS-6411, we need to catch the
> AccessControlException from the HDFS calls, and return NFS3ERR_PERM instead
> of NFS3ERR_IO for it.
> Another possible improvement is to have a single class/method for the common
> exception handling process, instead of repeating the same exception handling
> process in different NFS methods.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[ https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HDFS-6790: -- Status: Patch Available (was: Open) > DFSUtil Should Use configuration.getPassword for SSL passwords > -- > > Key: HDFS-6790 > URL: https://issues.apache.org/jira/browse/HDFS-6790 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Larry McCay > Attachments: HDFS-6790.patch, HDFS-6790.patch > > > As part of HADOOP-10904, DFSUtil should be changed to leverage the new method > on Configuration for acquiring known passwords for SSL. The getPassword > method will leverage the credential provider API and/or fallback to the clear > text value stored in ssl-server.xml. > This will provide an alternative to clear text passwords on disk while > maintaining backward compatibility for this behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[ https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HDFS-6790: -- Attachment: HDFS-6790.patch Removed invalid line in TestDFSUtil.testGetPassword() and attaching new patch. > DFSUtil Should Use configuration.getPassword for SSL passwords > -- > > Key: HDFS-6790 > URL: https://issues.apache.org/jira/browse/HDFS-6790 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Larry McCay > Attachments: HDFS-6790.patch, HDFS-6790.patch > > > As part of HADOOP-10904, DFSUtil should be changed to leverage the new method > on Configuration for acquiring known passwords for SSL. The getPassword > method will leverage the credential provider API and/or fallback to the clear > text value stored in ssl-server.xml. > This will provide an alternative to clear text passwords on disk while > maintaining backward compatibility for this behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[ https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HDFS-6790: -- Status: Open (was: Patch Available) Invalid line in test. > DFSUtil Should Use configuration.getPassword for SSL passwords > -- > > Key: HDFS-6790 > URL: https://issues.apache.org/jira/browse/HDFS-6790 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Larry McCay > Attachments: HDFS-6790.patch > > > As part of HADOOP-10904, DFSUtil should be changed to leverage the new method > on Configuration for acquiring known passwords for SSL. The getPassword > method will leverage the credential provider API and/or fallback to the clear > text value stored in ssl-server.xml. > This will provide an alternative to clear text passwords on disk while > maintaining backward compatibility for this behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6451) NFS should not return NFS3ERR_IO for AccessControlException
[ https://issues.apache.org/jira/browse/HDFS-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhiraj Butala updated HDFS-6451: - Attachment: HDFS-6451.003.patch Reattaching the patch with findbug warning addressed. > NFS should not return NFS3ERR_IO for AccessControlException > > > Key: HDFS-6451 > URL: https://issues.apache.org/jira/browse/HDFS-6451 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Brandon Li > Attachments: HDFS-6451.002.patch, HDFS-6451.003.patch, HDFS-6451.patch > > > As [~jingzhao] pointed out in HDFS-6411, we need to catch the > AccessControlException from the HDFS calls, and return NFS3ERR_PERM instead > of NFS3ERR_IO for it. > Another possible improvement is to have a single class/method for the common > exception handling process, instead of repeating the same exception handling > process in different NFS methods. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[
https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084128#comment-14084128
]
Hadoop QA commented on HDFS-6790:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659559/HDFS-6790.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDFSUtil
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7549//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7549//console
This message is automatically generated.
> DFSUtil Should Use configuration.getPassword for SSL passwords
> --
>
> Key: HDFS-6790
> URL: https://issues.apache.org/jira/browse/HDFS-6790
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Larry McCay
> Attachments: HDFS-6790.patch
>
>
> As part of HADOOP-10904, DFSUtil should be changed to leverage the new method
> on Configuration for acquiring known passwords for SSL. The getPassword
> method will leverage the credential provider API and/or fallback to the clear
> text value stored in ssl-server.xml.
> This will provide an alternative to clear text passwords on disk while
> maintaining backward compatibility for this behavior.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084122#comment-14084122
]
Hadoop QA commented on HDFS-6776:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659555/HDFS-6776.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7548//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7548//console
This message is automatically generated.
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroup
[jira] [Work started] (HDFS-6814) Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean
[
https://issues.apache.org/jira/browse/HDFS-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-6814 started by Uma Maheswara Rao G.
> Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as
> boolean
> -
>
> Key: HDFS-6814
> URL: https://issues.apache.org/jira/browse/HDFS-6814
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-6814.patch
>
>
> {code}
>
> dfs.namenode.list.encryption.zones.num.responses
> false
> When listing encryption zones, the maximum number of zones
> that will be returned in a batch. Fetching the list incrementally in
> batches improves namenode performance.
>
>
> {code}
> default value should be 100. Should be same as {code}public static final int
> DFS_NAMENODE_LIST_ENCRYPTION_ZONES_NUM_RESPONSES_DEFAULT = 100;{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HDFS-6814) Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean
[
https://issues.apache.org/jira/browse/HDFS-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084108#comment-14084108
]
Uma Maheswara Rao G edited comment on HDFS-6814 at 8/3/14 7:37 PM:
---
Noticed while running TestDistributedFileSystem,
{noformat}
java.lang.NumberFormatException: For input string: "false"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at java.lang.Integer.parseInt(Unknown Source)
at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1113)
at
org.apache.hadoop.hdfs.server.namenode.EncryptionZoneManager.(EncryptionZoneManager.java:75)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.(FSDirectory.java:231)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:880)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:752)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:925)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:291)
at
org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:146)
at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:869)
at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:707)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:378)
at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:359)
at
org.apache.hadoop.hdfs.TestDistributedFileSystem.testGetFileBlockStorageLocationsBatching(TestDistributedFileSystem.java:674)
{noformat}
was (Author: umamaheswararao):
Attached simple patch
> Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as
> boolean
> -
>
> Key: HDFS-6814
> URL: https://issues.apache.org/jira/browse/HDFS-6814
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-6814.patch
>
>
> {code}
>
> dfs.namenode.list.encryption.zones.num.responses
> false
> When listing encryption zones, the maximum number of zones
> that will be returned in a batch. Fetching the list incrementally in
> batches improves namenode performance.
>
>
> {code}
> default value should be 100. Should be same as {code}public static final int
> DFS_NAMENODE_LIST_ENCRYPTION_ZONES_NUM_RESPONSES_DEFAULT = 100;{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6814) Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean
[
https://issues.apache.org/jira/browse/HDFS-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084109#comment-14084109
]
Uma Maheswara Rao G commented on HDFS-6814:
---
Attached simple patch.
> Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as
> boolean
> -
>
> Key: HDFS-6814
> URL: https://issues.apache.org/jira/browse/HDFS-6814
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-6814.patch
>
>
> {code}
>
> dfs.namenode.list.encryption.zones.num.responses
> false
> When listing encryption zones, the maximum number of zones
> that will be returned in a batch. Fetching the list incrementally in
> batches improves namenode performance.
>
>
> {code}
> default value should be 100. Should be same as {code}public static final int
> DFS_NAMENODE_LIST_ENCRYPTION_ZONES_NUM_RESPONSES_DEFAULT = 100;{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6814) Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean
[
https://issues.apache.org/jira/browse/HDFS-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-6814:
--
Attachment: HDFS-6814.patch
Attached simple patch
> Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as
> boolean
> -
>
> Key: HDFS-6814
> URL: https://issues.apache.org/jira/browse/HDFS-6814
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-6814.patch
>
>
> {code}
>
> dfs.namenode.list.encryption.zones.num.responses
> false
> When listing encryption zones, the maximum number of zones
> that will be returned in a batch. Fetching the list incrementally in
> batches improves namenode performance.
>
>
> {code}
> default value should be 100. Should be same as {code}public static final int
> DFS_NAMENODE_LIST_ENCRYPTION_ZONES_NUM_RESPONSES_DEFAULT = 100;{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6814) Mistakenly dfs.namenode.list.encryption.zones.num.responses configured as boolean
Uma Maheswara Rao G created HDFS-6814:
-
Summary: Mistakenly
dfs.namenode.list.encryption.zones.num.responses configured as boolean
Key: HDFS-6814
URL: https://issues.apache.org/jira/browse/HDFS-6814
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
{code}
dfs.namenode.list.encryption.zones.num.responses
false
When listing encryption zones, the maximum number of zones
that will be returned in a batch. Fetching the list incrementally in
batches improves namenode performance.
{code}
default value should be 100. Should be same as {code}public static final int
DFS_NAMENODE_LIST_ENCRYPTION_ZONES_NUM_RESPONSES_DEFAULT = 100;{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6813) WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable with thead-safe.
[
https://issues.apache.org/jira/browse/HDFS-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084056#comment-14084056
]
Uma Maheswara Rao G commented on HDFS-6813:
---
I think from PositionedReadable doc, this looks reasonable to me. But I also
noticed FsInputStream also have the APIs with out synchronized. Also the read
api in DFSInputStream also not synchronized, but sure that was left with
synchronization with intention.
[~szetszwo], can you please confirm if there is any reason for not synchronized
and did not follow the PositionedReadable java doc? Thanks
> WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable
> with thead-safe.
> ---
>
> Key: HDFS-6813
> URL: https://issues.apache.org/jira/browse/HDFS-6813
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-6813.001.patch
>
>
> {{PositionedReadable}} definition requires the implementations for its
> interfaces should be thread-safe.
> OffsetUrlInputStream(WebHdfsFileSystem inputstream) doesn't implement these
> interfaces with tread-safe, this JIRA is to fix this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[ https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HDFS-6790: -- Status: Patch Available (was: Open) > DFSUtil Should Use configuration.getPassword for SSL passwords > -- > > Key: HDFS-6790 > URL: https://issues.apache.org/jira/browse/HDFS-6790 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Larry McCay > Attachments: HDFS-6790.patch > > > As part of HADOOP-10904, DFSUtil should be changed to leverage the new method > on Configuration for acquiring known passwords for SSL. The getPassword > method will leverage the credential provider API and/or fallback to the clear > text value stored in ssl-server.xml. > This will provide an alternative to clear text passwords on disk while > maintaining backward compatibility for this behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6790) DFSUtil Should Use configuration.getPassword for SSL passwords
[ https://issues.apache.org/jira/browse/HDFS-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HDFS-6790: -- Attachment: HDFS-6790.patch Patch to leverage Configuration.getPassword in order to provide an alternative to SSL passwords stored in clear text within ssl-server.xml or a side file - while maintaining backward compatibility. > DFSUtil Should Use configuration.getPassword for SSL passwords > -- > > Key: HDFS-6790 > URL: https://issues.apache.org/jira/browse/HDFS-6790 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Larry McCay > Attachments: HDFS-6790.patch > > > As part of HADOOP-10904, DFSUtil should be changed to leverage the new method > on Configuration for acquiring known passwords for SSL. The getPassword > method will leverage the credential provider API and/or fallback to the clear > text value stored in ssl-server.xml. > This will provide an alternative to clear text passwords on disk while > maintaining backward compatibility for this behavior. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-6776:
Attachment: HDFS-6776.003.patch
Version 003 to address findbugs issue.
> distcp from insecure cluster (source) to secure cluster (destination) doesn't
> work
> --
>
> Key: HDFS-6776
> URL: https://issues.apache.org/jira/browse/HDFS-6776
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.5.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
> HDFS-6776.003.patch
>
>
> Issuing distcp command at the secure cluster side, trying to copy stuff from
> insecure cluster to secure cluster, and see the following problem:
> {code}
> hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://:/tmp
> hdfs://:8020/tmp/tmptgt
> 14/07/30 20:06:19 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[webhdfs://:/tmp],
> targetPath=hdfs://:8020/tmp/tmptgt, targetPathExists=true}
> 14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
> :8032
> 14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
> 'ssl.client.truststore.location' has not been set, no TrustStore will be
> loaded
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Failed to get the token for hadoopuser,
> user=hadoopuser
> 14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
> at
> org.apache.hadoop
[jira] [Commented] (HDFS-6810) StorageReport array is initialized with wrong size in DatanodeDescriptor#getStorageReports
[
https://issues.apache.org/jira/browse/HDFS-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084003#comment-14084003
]
Hudson commented on HDFS-6810:
--
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1851 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1851/])
HDFS-6810. StorageReport array is initialized with wrong size in
DatanodeDescriptor#getStorageReports. (Contributed by szetszwo) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615381)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
> StorageReport array is initialized with wrong size in
> DatanodeDescriptor#getStorageReports
> --
>
> Key: HDFS-6810
> URL: https://issues.apache.org/jira/browse/HDFS-6810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ted Yu
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Fix For: 3.0.0, 2.6.0
>
> Attachments: h6810_20140803.patch
>
>
> Here is related code:
> {code}
> public StorageReport[] getStorageReports() {
> final StorageReport[] reports = new StorageReport[storageMap.size()];
> {code}
> Other methods use the following construct:
> {code}
> synchronized (storageMap) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6810) StorageReport array is initialized with wrong size in DatanodeDescriptor#getStorageReports
[
https://issues.apache.org/jira/browse/HDFS-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083979#comment-14083979
]
Hudson commented on HDFS-6810:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1826 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1826/])
HDFS-6810. StorageReport array is initialized with wrong size in
DatanodeDescriptor#getStorageReports. (Contributed by szetszwo) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615381)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
> StorageReport array is initialized with wrong size in
> DatanodeDescriptor#getStorageReports
> --
>
> Key: HDFS-6810
> URL: https://issues.apache.org/jira/browse/HDFS-6810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ted Yu
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Fix For: 3.0.0, 2.6.0
>
> Attachments: h6810_20140803.patch
>
>
> Here is related code:
> {code}
> public StorageReport[] getStorageReports() {
> final StorageReport[] reports = new StorageReport[storageMap.size()];
> {code}
> Other methods use the following construct:
> {code}
> synchronized (storageMap) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6813) WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable with thead-safe.
[
https://issues.apache.org/jira/browse/HDFS-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083956#comment-14083956
]
Hadoop QA commented on HDFS-6813:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659533/HDFS-6813.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSZKFailoverController
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7546//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7546//console
This message is automatically generated.
> WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable
> with thead-safe.
> ---
>
> Key: HDFS-6813
> URL: https://issues.apache.org/jira/browse/HDFS-6813
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-6813.001.patch
>
>
> {{PositionedReadable}} definition requires the implementations for its
> interfaces should be thread-safe.
> OffsetUrlInputStream(WebHdfsFileSystem inputstream) doesn't implement these
> interfaces with tread-safe, this JIRA is to fix this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6451) NFS should not return NFS3ERR_IO for AccessControlException
[
https://issues.apache.org/jira/browse/HDFS-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083953#comment-14083953
]
Hadoop QA commented on HDFS-6451:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659540/HDFS-6451.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7547//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7547//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7547//console
This message is automatically generated.
> NFS should not return NFS3ERR_IO for AccessControlException
>
>
> Key: HDFS-6451
> URL: https://issues.apache.org/jira/browse/HDFS-6451
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Reporter: Brandon Li
> Attachments: HDFS-6451.002.patch, HDFS-6451.patch
>
>
> As [~jingzhao] pointed out in HDFS-6411, we need to catch the
> AccessControlException from the HDFS calls, and return NFS3ERR_PERM instead
> of NFS3ERR_IO for it.
> Another possible improvement is to have a single class/method for the common
> exception handling process, instead of repeating the same exception handling
> process in different NFS methods.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6810) StorageReport array is initialized with wrong size in DatanodeDescriptor#getStorageReports
[
https://issues.apache.org/jira/browse/HDFS-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083952#comment-14083952
]
Hudson commented on HDFS-6810:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #632 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/632/])
HDFS-6810. StorageReport array is initialized with wrong size in
DatanodeDescriptor#getStorageReports. (Contributed by szetszwo) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1615381)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
> StorageReport array is initialized with wrong size in
> DatanodeDescriptor#getStorageReports
> --
>
> Key: HDFS-6810
> URL: https://issues.apache.org/jira/browse/HDFS-6810
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Ted Yu
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Fix For: 3.0.0, 2.6.0
>
> Attachments: h6810_20140803.patch
>
>
> Here is related code:
> {code}
> public StorageReport[] getStorageReports() {
> final StorageReport[] reports = new StorageReport[storageMap.size()];
> {code}
> Other methods use the following construct:
> {code}
> synchronized (storageMap) {
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6803) Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
[
https://issues.apache.org/jira/browse/HDFS-6803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083949#comment-14083949
]
Steve Loughran commented on HDFS-6803:
--
This is fun, stack's just opened up a whole new bag of inconsistencies.
h2. Consistency with actual file data & metadata
We should state that changes to a file (length, contents, existence, perms) may
not be visible to an open stream; if they do become visible there are no
guarantees when those changes become visible. That could include partway
through a readFully operation -this cannot guaranteed to be atomic.
h2. Isolation of pread operations
When a pread is in progress, should that change be visible in {{getPos()}}?
# If not, the method will need to be made {{synchronized}} on all
implementations (it isn't right now; I checked). I
# If it can be visible, then we could pull the {{synchronized}} marker off
some implementations and remove that as a lock point.
h2. Failure Modes in concurrent/serialized operations
One problem with concurrency on read+pread is something I hadn't thought of
before: on any failure of a pread, the pos value must be reset to the previous
one. Everything appears to do this; the test would be
{code}
read()
try{
read(EOF+2)
} catch (IOException) {
}
assertTrue(getPos()<=EOF)
read()
{code}
The second {{read()}} would succeed/return -1 depending on the position, and
not an {{EOFException}}. The same outcome must happen for a negative pread
attempt.
If someone were to add this to {{AbstractContractSeekTest}} it'd get picked up
by all the implementations and we could see what happens.
Looking at the standard impl, it does seek() back in a finally block -but if
there is an exception in the read(), then a subsequent exception in the final
seek() would lose that. I think it should be reworked to catch any IOE in the
read operation and do an exception-swallowing seek-back in this case. Or just
do it for EOFException now that Hadoop 2.5+ has all the standard filesystems
throwing EOFException consistently.
> Documenting DFSClient#DFSInputStream expectations reading and preading in
> concurrent context
>
>
> Key: HDFS-6803
> URL: https://issues.apache.org/jira/browse/HDFS-6803
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Affects Versions: 2.4.1
>Reporter: stack
> Attachments: DocumentingDFSClientDFSInputStream (1).pdf
>
>
> Reviews of the patch posted the parent task suggest that we be more explicit
> about how DFSIS is expected to behave when being read by contending threads.
> It is also suggested that presumptions made internally be made explicit
> documenting expectations.
> Before we put up a patch we've made a document of assertions we'd like to
> make into tenets of DFSInputSteam. If agreement, we'll attach to this issue
> a patch that weaves the assumptions into DFSIS as javadoc and class comments.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6451) NFS should not return NFS3ERR_IO for AccessControlException
[ https://issues.apache.org/jira/browse/HDFS-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083946#comment-14083946 ] Abhiraj Butala commented on HDFS-6451: -- Forgot to mention, I also cleaned up some white spaces in RpcProgramNf3.java. Please forgive me for that. :) > NFS should not return NFS3ERR_IO for AccessControlException > > > Key: HDFS-6451 > URL: https://issues.apache.org/jira/browse/HDFS-6451 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Brandon Li > Attachments: HDFS-6451.002.patch, HDFS-6451.patch > > > As [~jingzhao] pointed out in HDFS-6411, we need to catch the > AccessControlException from the HDFS calls, and return NFS3ERR_PERM instead > of NFS3ERR_IO for it. > Another possible improvement is to have a single class/method for the common > exception handling process, instead of repeating the same exception handling > process in different NFS methods. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6451) NFS should not return NFS3ERR_IO for AccessControlException
[ https://issues.apache.org/jira/browse/HDFS-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhiraj Butala updated HDFS-6451: - Attachment: HDFS-6451.002.patch Attaching a patch which addresses the review comments by [~brandonli]. Added tests for all the handlers in TestRpcProgramNfs3.java. Kept the tests generic, so they can be extended in future to include other tests (various corner cases, other NFS3ERR* messages, etc). While testing read() I hit HDFS-6582. I have made a note of this and commented that specific test for now. Let me know if there are any suggestions. Thanks! > NFS should not return NFS3ERR_IO for AccessControlException > > > Key: HDFS-6451 > URL: https://issues.apache.org/jira/browse/HDFS-6451 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Brandon Li > Attachments: HDFS-6451.002.patch, HDFS-6451.patch > > > As [~jingzhao] pointed out in HDFS-6411, we need to catch the > AccessControlException from the HDFS calls, and return NFS3ERR_PERM instead > of NFS3ERR_IO for it. > Another possible improvement is to have a single class/method for the common > exception handling process, instead of repeating the same exception handling > process in different NFS methods. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6813) WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable with thead-safe.
[
https://issues.apache.org/jira/browse/HDFS-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-6813:
-
Attachment: HDFS-6813.001.patch
> WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable
> with thead-safe.
> ---
>
> Key: HDFS-6813
> URL: https://issues.apache.org/jira/browse/HDFS-6813
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-6813.001.patch
>
>
> {{PositionedReadable}} definition requires the implementations for its
> interfaces should be thread-safe.
> OffsetUrlInputStream(WebHdfsFileSystem inputstream) doesn't implement these
> interfaces with tread-safe, this JIRA is to fix this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6813) WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable with thead-safe.
[
https://issues.apache.org/jira/browse/HDFS-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-6813:
-
Status: Patch Available (was: Open)
> WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable
> with thead-safe.
> ---
>
> Key: HDFS-6813
> URL: https://issues.apache.org/jira/browse/HDFS-6813
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-6813.001.patch
>
>
> {{PositionedReadable}} definition requires the implementations for its
> interfaces should be thread-safe.
> OffsetUrlInputStream(WebHdfsFileSystem inputstream) doesn't implement these
> interfaces with tread-safe, this JIRA is to fix this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6813) WebHdfsFileSystem#OffsetUrlInputStream should implement PositionedReadable with thead-safe.
Yi Liu created HDFS-6813:
Summary: WebHdfsFileSystem#OffsetUrlInputStream should implement
PositionedReadable with thead-safe.
Key: HDFS-6813
URL: https://issues.apache.org/jira/browse/HDFS-6813
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.6.0
Reporter: Yi Liu
Assignee: Yi Liu
{{PositionedReadable}} definition requires the implementations for its
interfaces should be thread-safe.
OffsetUrlInputStream(WebHdfsFileSystem inputstream) doesn't implement these
interfaces with tread-safe, this JIRA is to fix this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6782) Improve FS editlog logSync
[
https://issues.apache.org/jira/browse/HDFS-6782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14083903#comment-14083903
]
Hadoop QA commented on HDFS-6782:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659525/HDFS-6782.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7545//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7545//console
This message is automatically generated.
> Improve FS editlog logSync
> --
>
> Key: HDFS-6782
> URL: https://issues.apache.org/jira/browse/HDFS-6782
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.4.1
>Reporter: Yi Liu
>Assignee: Yi Liu
> Attachments: HDFS-6782.001.patch, HDFS-6782.002.patch
>
>
> In NN, it uses a double buffer (bufCurrent, bufReady) for log sync,
> bufCurrent it to buffer new coming edit ops and bufReady is for flushing.
> This's efficient. When flush is ongoing, and bufCurrent is full, NN goes to
> force log sync, and all new Ops are blocked (since force log sync is
> protected by FSNameSystem write lock). After the flush finished, the new Ops
> are still blocked, but actually at this time, bufCurrent is free and Ops can
> go ahead and write to the buffer. The following diagram shows the detail.
> This JIRA is for this improvement. Thanks [~umamaheswararao] for confirming
> this issue.
> {code}
> edit1(txid1) -- write to bufCurrent logSync - (swap
> buffer)flushing ---
> edit2(txid2) -- write to bufCurrent logSync - waiting
> ---
> edit3(txid3) -- write to bufCurrent logSync - waiting
> ---
> edit4(txid4) -- write to bufCurrent logSync - waiting
> ---
> edit5(txid5) -- write to bufCurrent --full-- force sync - waiting
> ---
> edit6(txid6) -- blocked
> ...
> editn(txidn) -- blocked
> {code}
> After the flush, it becomes
> {code}
> edit1(txid1) -- write to bufCurrent logSync - finished
>
> edit2(txid2) -- write to bufCurrent logSync - flushing
> ---
> edit3(txid3) -- write to bufCurrent logSync - waiting
> ---
> edit4(txid4) -- write to bufCurrent logSync - waiting
> ---
> edit5(txid5) -- write to bufCurrent --full-- force sync - waiting
> ---
> edit6(txid6) -- blocked
> ...
> editn(txidn) -- blocked
> {code}
> After edit1 finished, bufCurrent is free, and the thread which flushes txid2
> will also flushes txid3-5, so we should return from the force sync of edit5
> and FSNamesystem write lock will be freed (Don't worry that edit5 Op will
> return, since there will be a normal logSync after the force logSync and
> there will wait for sync finished). This is the idea of this JIRA.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
