[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133275#comment-13133275 ] Konstantin Boudnik commented on HDFS-2452: -- +1 looks good. As for the naming: [JIRA].patch stands for trunk patch, IMO. Any branch designations can be add as qualifiers. Thanks > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0, 0.23.0, 0.24.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch, HDFS-2452-Trunk-src-fix.patch, > HDFS-2452-Trunk.patch, HDFS-2452.patch, HDFS-2452.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2452: -- Attachment: HDFS-2452.patch > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0, 0.23.0, 0.24.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch, HDFS-2452-Trunk-src-fix.patch, > HDFS-2452-Trunk.patch, HDFS-2452.patch, HDFS-2452.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2452: -- Affects Version/s: 0.24.0 0.23.0 > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0, 0.23.0, 0.24.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch, HDFS-2452-Trunk-src-fix.patch, > HDFS-2452-Trunk.patch, HDFS-2452.patch, HDFS-2452.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[
https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133270#comment-13133270
]
Uma Maheswara Rao G commented on HDFS-2452:
---
Thanks for the review!
{quote}
# + new Daemon(datanode.threadGroup, new DataXceiver(s, datanode,
this)).start(); is too long. Should be longer than 80 chars.
# you don't need to change the name for every new patch: JIRA will make sure
that the latest one is shown as 'enabled'. Custom naming is confusing, actually.
{quote}
wrapped it.
I have been observing in other JIRAs with this namings;). To avoid confusions,
i just named it with JIRA id.
Thanks
Uma
> OutOfMemoryError in DataXceiverServer takes down the DataNode
> -
>
> Key: HDFS-2452
> URL: https://issues.apache.org/jira/browse/HDFS-2452
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.22.0
>Reporter: Konstantin Shvachko
>Assignee: Uma Maheswara Rao G
> Fix For: 0.22.0
>
> Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch,
> HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch,
> HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch,
> HDFS-2452-22branch_with-around_.patch, HDFS-2452-Trunk-src-fix.patch,
> HDFS-2452-Trunk.patch, HDFS-2452.patch
>
>
> OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn
> a new data transfer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[
https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133261#comment-13133261
]
Konstantin Boudnik commented on HDFS-2452:
--
Coupla nits:
- {{+new Daemon(datanode.threadGroup, new DataXceiver(s, datanode,
this)).start();}} is too long. Should be longer than 80 chars.
- you don't need to change the name for every new patch: JIRA will make sure
that the latest one is shown as 'enabled'. Custom naming is confusing, actually.
- nice catch with IS' getter: I've missed it completely
Looks good otherwise.
> OutOfMemoryError in DataXceiverServer takes down the DataNode
> -
>
> Key: HDFS-2452
> URL: https://issues.apache.org/jira/browse/HDFS-2452
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.22.0
>Reporter: Konstantin Shvachko
>Assignee: Uma Maheswara Rao G
> Fix For: 0.22.0
>
> Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch,
> HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch,
> HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch,
> HDFS-2452-22branch_with-around_.patch, HDFS-2452-Trunk-src-fix.patch,
> HDFS-2452-Trunk.patch, HDFS-2452.patch
>
>
> OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn
> a new data transfer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2452: -- Attachment: HDFS-2452-Trunk.patch Here is a patch for trunk with test. 1. In trunk code, InputStream getting logic has been moced to DataXceiver constructor. So, we need to stub getInputStream also.Updated with this change in test. Thanks Uma > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch, HDFS-2452-Trunk-src-fix.patch, > HDFS-2452-Trunk.patch, HDFS-2452.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133254#comment-13133254 ] Konstantin Boudnik commented on HDFS-2452: -- Uma, actually there's no need for a separate patch: FI test will be picked up automatically once someone (likely to be me though) get AOP fixed in maven env. > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch, HDFS-2452-Trunk-src-fix.patch, > HDFS-2452.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2476) More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks
[ https://issues.apache.org/jira/browse/HDFS-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Nykiel updated HDFS-2476: Attachment: hashStructures.patch-4 > More CPU efficient data structure for > under-replicated/over-replicated/invalidate blocks > > > Key: HDFS-2476 > URL: https://issues.apache.org/jira/browse/HDFS-2476 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tomasz Nykiel >Assignee: Tomasz Nykiel > Attachments: hashStructures.patch, hashStructures.patch-2, > hashStructures.patch-3, hashStructures.patch-4 > > > This patch introduces two hash data structures for storing under-replicated, > over-replicated and invalidated blocks. > 1. LightWeightHashSet > 2. LightWeightLinkedSet > Currently in all these cases we are using java.util.TreeSet which adds > unnecessary overhead. > The main bottlenecks addressed by this patch are: > -cluster instability times, when these queues (especially under-replicated) > tend to grow quite drastically, > -initial cluster startup, when the queues are initialized, after leaving > safemode, > -block reports, > -explicit acks for block addition and deletion > 1. The introduced structures are CPU-optimized. > 2. They shrink and expand according to current capacity. > 3. Add/contains/delete ops are performed in O(1) time (unlike current log n > for TreeSet). > 4. The sets are equipped with fast access methods for polling a number of > elements (get+remove), which are used for handling the queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2012) Recurring failure of TestBalancer due to incorrect treatment of nodes whose utilization equals avgUtilization.
[ https://issues.apache.org/jira/browse/HDFS-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133248#comment-13133248 ] Uma Maheswara Rao G commented on HDFS-2012: --- Here is the Jira for handling it. HDFS-2491 > Recurring failure of TestBalancer due to incorrect treatment of nodes whose > utilization equals avgUtilization. > -- > > Key: HDFS-2012 > URL: https://issues.apache.org/jira/browse/HDFS-2012 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer, test >Affects Versions: 0.22.0 >Reporter: Aaron T. Myers >Assignee: Uma Maheswara Rao G >Priority: Blocker > Fix For: 0.22.0, 0.24.0 > > Attachments: HDFS-2012-0.22Branch.patch, HDFS-2012-Trunk.patch, > HDFS-2012.patch, TestBalancerLog.html > > > This has been failing on Hudson for the last two builds and fails on my local > box as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2491) TestBalancer can fail when datanode utilization and avgUtilization is exactly same.
TestBalancer can fail when datanode utilization and avgUtilization is exactly same. --- Key: HDFS-2491 URL: https://issues.apache.org/jira/browse/HDFS-2491 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0, 0.23.0, 0.24.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Stack Trace: junit.framework.AssertionFailedError: 127.0.0.1:60986is not an underUtilized node: utilization=22.0 avgUtilization=22.0 threshold=10.0 at org.apache.hadoop.hdfs.server.balancer.Balancer.initNodes(Balancer.java:1014) at org.apache.hadoop.hdfs.server.balancer.Balancer.initNodes(Balancer.java:953) at org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:1502) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:247) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.test(TestBalancer.java:234) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.twoNodeTest(TestBalancer.java:312) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.__CLR2_4_39j3j5b10ou(TestBalancer.java:328) at org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0(TestBalancer.java:324) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2452: -- Attachment: HDFS-2452-Trunk-src-fix.patch Sorry for coming late, Thanks a lot for making the patch on trunk as well! Cos, Patch mostly looks good. But one problem is IOUtils.socketClose() repeated in the patch. Because socket close already handle in try block itself for IOEceptions. To avoid confusions i just made it sync with 22 as well. Here is a patch for trunk (only src changes). I will file separate JIRA for faultinjection test. Thanks, Uma > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch, HDFS-2452-Trunk-src-fix.patch, > HDFS-2452.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2452: - Attachment: HDFS-2452.patch Here's a patch for trunk. Something wrong with my ant installation or something and maven complains that it can't run the build. Thus, I can't verify it right now - will do later. The patch also contains new test and instrumentation but it won't be verified until AOP is fixed in trunk. > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch, HDFS-2452.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133237#comment-13133237 ] Tsz Wo (Nicholas), SZE commented on HDFS-2316: -- *Thanks everyone for looking at the webhdfs API.* I will update the doc accordingly. Sorry the json types and error responses are missing. Below are some quick responses: > 4. case sensitivity - make the parameters lower case rather than have the > filter convert them since pathname and the user name should not be converted. Only the parameter names are case insensitive. The parameter values are case sensitive except for op values and boolean values. > Permission masks are currently octal in webhdfs and symbolic in hoop. IMO, it > should make sense to support both. I thought about adding chmod style symbolic permission. However, it does not make sense in webhdfs SETPERMISSION since it only sets absolute permission (e.g. "u=rwx,go=rx", which equals 755.) The relative permissions (e.g. "go-w") won't work. Hoop uses ls output style for setting permission (e.g. rwxr-xr-x). This seems uncommon. > File ranges, webhdfs uses HTTP 'content-ranges' header, ... I have implemented OPEN with offset and length parameters but decide to change it since it is not following the http spec. Won't it cause problems if webhdfs does follow the http spec? > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: WebHdfsAPI20111020.pdf > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2476) More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks
[ https://issues.apache.org/jira/browse/HDFS-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133209#comment-13133209 ] Hadoop QA commented on HDFS-2476: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500263/hashStructures.patch-3 against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDataNodeExit org.apache.hadoop.hdfs.TestDFSRename org.apache.hadoop.security.TestRefreshUserMappings org.apache.hadoop.hdfs.server.datanode.TestRefreshNamenodes org.apache.hadoop.hdfs.TestLeaseRecovery org.apache.hadoop.fs.loadGenerator.TestLoadGenerator org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner org.apache.hadoop.hdfs.server.blockmanagement.TestNodeCount org.apache.hadoop.fs.TestGlobPaths org.apache.hadoop.fs.TestResolveHdfsSymlink org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.fs.TestUrlStreamHandler org.apache.hadoop.fs.viewfs.TestViewFsFileStatusHdfs org.apache.hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.fs.TestFcHdfsPermission org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.TestListFilesInFileContext org.apache.hadoop.hdfs.server.datanode.TestBlockReplacement org.apache.hadoop.hdfs.TestListPathServlet org.apache.hadoop.hdfs.TestLocalDFS org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.TestDataTransferProtocol org.apache.hadoop.hdfs.TestFileCreationDelete org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.hdfs.server.datanode.TestWriteToReplica org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks org.apache.hadoop.hdfs.TestDFSMkdirs org.apache.hadoop.fs.TestFcHdfsCreateMkdir org.apache.hadoop.security.TestPermission org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs org.apache.hadoop.fs.TestHDFSFileContextMainOperations org.apache.hadoop.hdfs.server.datanode.TestDiskError org.apache.hadoop.hdfs.server.datanode.TestTransferRbw org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.server.datanode.TestDatanodeJsp org.apache.hadoop.hdfs.TestRenameWhileOpen org.apache.hadoop.hdfs.TestDFSUpgradeFromImage org.apache.hadoop.fs.viewfs.TestViewFsHdfs org.apache.hadoop.hdfs.TestDatanodeReport org.apache.hadoop.hdfs.TestHDFSFileSystemContract org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner org.apache.hadoop.hdfs.TestDFSPermission org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.fs.TestFcHdfsSymlink org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.hdfs.TestReadWhileWriting org.apache.hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork
[jira] [Commented] (HDFS-2489) Move commands Finalize and Register out of DatanodeCommand class.
[ https://issues.apache.org/jira/browse/HDFS-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133197#comment-13133197 ] Jitendra Nath Pandey commented on HDFS-2489: # Javadoc for FinalizeCommand and RegisterCommand refers to BlockCommand. # @Override in a few methods of FinalizeCommand could be added. > Move commands Finalize and Register out of DatanodeCommand class. > - > > Key: HDFS-2489 > URL: https://issues.apache.org/jira/browse/HDFS-2489 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0, 0.24.0 > Environment: There are other subclasses available in separate files. > These commands should be moved to separate files as well. >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2489.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2334) Add Closeable to JournalManager
[ https://issues.apache.org/jira/browse/HDFS-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133196#comment-13133196 ] Jitendra Nath Pandey commented on HDFS-2334: In JournalAndStream#close, stream may already be closed and may be null. You probably need to check for null for stream. > Add Closeable to JournalManager > --- > > Key: HDFS-2334 > URL: https://issues.apache.org/jira/browse/HDFS-2334 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: HDFS-2334.diff, HDFS-2334.diff > > > A JournalManager may take hold of resources for the duration of their > lifetime. This isn't the case at the moment for FileJournalManager, but > BookKeeperJournalManager will, and it's conceivable that FileJournalManager > could take a lock on a directory etc. > This JIRA is to add Closeable to JournalManager so that these resources can > be cleaned up when FSEditLog is closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133189#comment-13133189 ] Alejandro Abdelnur commented on HDFS-2178: -- Regarding the getHandle, if I understood correctly Owen's idea (via Sanjay) is that for create/append we first do the put/post call with op=create/append and no data; and the returned Location's URL would be a URL containing the same query string plus an opaque extra parameter that flags the server that the request is for uploading data (as opposed to get a handle). > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.23.0 > > Attachments: HDFS-2178.patch, HDFSoverHTTP-API.html, HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133175#comment-13133175 ] Milind Bhandarkar commented on HDFS-2316: - Guys, is there a documentation for webhdfs APIs that I can read somewhere ? (A good advice for producing human readable documentation for webservices can be found here: http://answers.oreilly.com/topic/1390-how-to-document-restful-web-services/). +1 to Nathan's suggestion for versioning the API. +1 to Alejandro's suggestion for embedding byte-ranges in the URL itself. > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: WebHdfsAPI20111020.pdf > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2476) More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks
[ https://issues.apache.org/jira/browse/HDFS-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Nykiel updated HDFS-2476: Status: Patch Available (was: Open) > More CPU efficient data structure for > under-replicated/over-replicated/invalidate blocks > > > Key: HDFS-2476 > URL: https://issues.apache.org/jira/browse/HDFS-2476 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tomasz Nykiel >Assignee: Tomasz Nykiel > Attachments: hashStructures.patch, hashStructures.patch-2, > hashStructures.patch-3 > > > This patch introduces two hash data structures for storing under-replicated, > over-replicated and invalidated blocks. > 1. LightWeightHashSet > 2. LightWeightLinkedSet > Currently in all these cases we are using java.util.TreeSet which adds > unnecessary overhead. > The main bottlenecks addressed by this patch are: > -cluster instability times, when these queues (especially under-replicated) > tend to grow quite drastically, > -initial cluster startup, when the queues are initialized, after leaving > safemode, > -block reports, > -explicit acks for block addition and deletion > 1. The introduced structures are CPU-optimized. > 2. They shrink and expand according to current capacity. > 3. Add/contains/delete ops are performed in O(1) time (unlike current log n > for TreeSet). > 4. The sets are equipped with fast access methods for polling a number of > elements (get+remove), which are used for handling the queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2476) More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks
[ https://issues.apache.org/jira/browse/HDFS-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Nykiel updated HDFS-2476: Attachment: hashStructures.patch-3 > More CPU efficient data structure for > under-replicated/over-replicated/invalidate blocks > > > Key: HDFS-2476 > URL: https://issues.apache.org/jira/browse/HDFS-2476 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tomasz Nykiel >Assignee: Tomasz Nykiel > Attachments: hashStructures.patch, hashStructures.patch-2, > hashStructures.patch-3 > > > This patch introduces two hash data structures for storing under-replicated, > over-replicated and invalidated blocks. > 1. LightWeightHashSet > 2. LightWeightLinkedSet > Currently in all these cases we are using java.util.TreeSet which adds > unnecessary overhead. > The main bottlenecks addressed by this patch are: > -cluster instability times, when these queues (especially under-replicated) > tend to grow quite drastically, > -initial cluster startup, when the queues are initialized, after leaving > safemode, > -block reports, > -explicit acks for block addition and deletion > 1. The introduced structures are CPU-optimized. > 2. They shrink and expand according to current capacity. > 3. Add/contains/delete ops are performed in O(1) time (unlike current log n > for TreeSet). > 4. The sets are equipped with fast access methods for polling a number of > elements (get+remove), which are used for handling the queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133173#comment-13133173 ] Milind Bhandarkar commented on HDFS-2316: - @Thejas, webhdfs:// is the scheme recognized by FileSystem.get in Hadoop. (Same thing as hftp://, which uses http protocol, but hftp is the file system impl.) > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: WebHdfsAPI20111020.pdf > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2012) Recurring failure of TestBalancer due to incorrect treatment of nodes whose utilization equals avgUtilization.
[ https://issues.apache.org/jira/browse/HDFS-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133150#comment-13133150 ] Konstantin Shvachko commented on HDFS-2012: --- Yes, we missed it. Open another jira, please. > Recurring failure of TestBalancer due to incorrect treatment of nodes whose > utilization equals avgUtilization. > -- > > Key: HDFS-2012 > URL: https://issues.apache.org/jira/browse/HDFS-2012 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer, test >Affects Versions: 0.22.0 >Reporter: Aaron T. Myers >Assignee: Uma Maheswara Rao G >Priority: Blocker > Fix For: 0.22.0, 0.24.0 > > Attachments: HDFS-2012-0.22Branch.patch, HDFS-2012-Trunk.patch, > HDFS-2012.patch, TestBalancerLog.html > > > This has been failing on Hudson for the last two builds and fails on my local > box as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2489) Move commands Finalize and Register out of DatanodeCommand class.
[ https://issues.apache.org/jira/browse/HDFS-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133142#comment-13133142 ] Hadoop QA commented on HDFS-2489: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500243/HDFS-2489.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.server.balancer.TestBalancer +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1410//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1410//console This message is automatically generated. > Move commands Finalize and Register out of DatanodeCommand class. > - > > Key: HDFS-2489 > URL: https://issues.apache.org/jira/browse/HDFS-2489 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0, 0.24.0 > Environment: There are other subclasses available in separate files. > These commands should be moved to separate files as well. >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2489.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133134#comment-13133134 ] Thejas M Nair commented on HDFS-2316: - bq. for scheme - i don't think we should use a prexisting scheme name. Nfs community has used webnfs as the scheme for accessing nfs over http. The plan is to support calls over HTTP, so I think it is better to keep that clear for the users. Are there any plans of supporting non http operations ? If not, I don't see any benefit of having a 'webhdfs' scheme. > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: WebHdfsAPI20111020.pdf > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133129#comment-13133129 ] Alejandro Abdelnur commented on HDFS-2316: -- Thanks Sanjay. A couple of follow up issues in the current API: * Permission masks are currently octal in webhdfs and symbolic in hoop. IMO, it should make sense to support both. * File ranges, webhdfs uses HTTP 'content-ranges' header, hoop uses 2 query string params offset= & len=. In webhdfs, except for this type of requests, for all other request the URL itself fully describes what is being requested. Because webhdfs uses the HTTP 'content-ranges' header a URL is not sufficient to specify the desired range. With Hoop approach a URL self-contains the desired range, making it easier to use from libraries and scripts. > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: WebHdfsAPI20111020.pdf > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2362) More Improvements on NameNode Scalability
[ https://issues.apache.org/jira/browse/HDFS-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Nykiel updated HDFS-2362: Description: This jira acts as an umbrella jira to track all the improvements we've done recently to improve Namenode's performance, responsiveness, and hence scalability. Those improvements include: 1. Incremental block reports (HDFS-395) 2. BlockManager.reportDiff optimization for processing block reports (HDFS-2477) 3. Upgradable lock to allow simutaleous read operation while reportDiff is in progress in processing block reports (HDFS-2490) 4. More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks (HDFS-2476) 5. Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock 6. Support variable block sizes 7. Release RPC handlers while waiting for edit log is synced to disk 8. Reduce network traffic pressure to the master rack where NN is located by lowering read priority of the replicas on the rack 9. A standalone KeepAlive heartbeat thread 10. Reduce Multiple traversals of path directory to one for most namespace manipulations 11. Move logging out of write lock section. was: This jira acts as an umbrella jira to track all the improvements we've done recently to improve Namenode's performance, responsiveness, and hence scalability. Those improvements include: 1. Incremental block reports (HDFS-395) 2. BlockManager.reportDiff optimization for processing block reports (HDFS-2477) 3. Upgradable lock to allow simutaleous read operation while reportDiff is in progress in processing block reports 4. More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks (HDFS-2476) 5. Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock 6. Support variable block sizes 7. Release RPC handlers while waiting for edit log is synced to disk 8. Reduce network traffic pressure to the master rack where NN is located by lowering read priority of the replicas on the rack 9. A standalone KeepAlive heartbeat thread 10. Reduce Multiple traversals of path directory to one for most namespace manipulations 11. Move logging out of write lock section. > More Improvements on NameNode Scalability > - > > Key: HDFS-2362 > URL: https://issues.apache.org/jira/browse/HDFS-2362 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang > > This jira acts as an umbrella jira to track all the improvements we've done > recently to improve Namenode's performance, responsiveness, and hence > scalability. Those improvements include: > 1. Incremental block reports (HDFS-395) > 2. BlockManager.reportDiff optimization for processing block reports > (HDFS-2477) > 3. Upgradable lock to allow simutaleous read operation while reportDiff is in > progress in processing block reports (HDFS-2490) > 4. More CPU efficient data structure for > under-replicated/over-replicated/invalidate blocks (HDFS-2476) > 5. Increase granularity of write operations in ReplicationMonitor thus > reducing contention for write lock > 6. Support variable block sizes > 7. Release RPC handlers while waiting for edit log is synced to disk > 8. Reduce network traffic pressure to the master rack where NN is located by > lowering read priority of the replicas on the rack > 9. A standalone KeepAlive heartbeat thread > 10. Reduce Multiple traversals of path directory to one for most namespace > manipulations > 11. Move logging out of write lock section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2490) Upgradable lock to allow simutaleous read operation while reportDiff is in progress in processing block reports
[ https://issues.apache.org/jira/browse/HDFS-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomasz Nykiel updated HDFS-2490: Attachment: FSNamesystemLock.java > Upgradable lock to allow simutaleous read operation while reportDiff is in > progress in processing block reports > --- > > Key: HDFS-2490 > URL: https://issues.apache.org/jira/browse/HDFS-2490 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tomasz Nykiel >Assignee: Tomasz Nykiel > Attachments: FSNamesystemLock.java > > > Currently, FSNamesystem operations are protected by a single > ReentrantReadWriteLock, which allows for having multiple concurrent readers > to perform reads, and a single writer to perform writes. There are, however, > operations whose execution has primarily reading nature, but occasionally > they write. > The finest example is processing block reports - currently the entire > processing is done under writeLock(). With HDFS-395 (explicit deletion acks), > processing a block report is primarily a read operation (reportDiff()) after > which only very few blocks need to be updated. In fact, we noticed this > number to be very low, or even zero blocks. > It would be desirable to have an upgradeable read lock, which would allow for > performing other reads during the first "read" part of reportDiff() (and > possibly other operations. > We implemented such mechanism, which provides writeLock(), readLock(), > upgradeableReadLock, upgradeLock(), and downgradeLock(). I achieved this be > emloying two ReentrantReadWriteLock's - one protects writes (lock1), the > other one reads (lock2). > Hence, we have: > writeLock() > lock1.writeLock().lock() > lock2.writeLock().lock() > readLock() > lock2.readLock().lock() > upgradeableReadLock() > lock1.writeLock().lock() > upgrade() > lock2.writeLock().lock() > -- > Hence a writeLock() is essentially equivalent to upgradeableLock()+upgrade() > - two writeLocks are mutually exclusive because of lock1.writeLock > - a writeLock and upgradeableLock are mutually exclusive as above > - readLock is mutually exclusive with upgradeableLock()+upgrade() OR > writeLock because of lock2.writeLock > - readLock() + writeLock() causes a deadlock, the same as currently > - writeLock() + readLock() does not cause deadlocks > -- > I am conviced to the soundness of this mechanism. > The overhead comes from having two locks, and in particular, writes need to > acquire both of them. > We deployed this feature, we used the upgradeableLock() ONLY for processing > reports. > Our initial, but not exhaustive experiments have shown that it had a very > detrimental effect on the NN throughput - writes were taking up to twice as > long. > This is very unexpected, and hard to explain by only the overhead of > acquiring additional lock for writes. > I would like to ask for input, as maybe I am missing some fundamental problem > here. > I am attaching a java class which implements this locking mechanism. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2490) Upgradable lock to allow simutaleous read operation while reportDiff is in progress in processing block reports
Upgradable lock to allow simutaleous read operation while reportDiff is in progress in processing block reports --- Key: HDFS-2490 URL: https://issues.apache.org/jira/browse/HDFS-2490 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tomasz Nykiel Assignee: Tomasz Nykiel Attachments: FSNamesystemLock.java Currently, FSNamesystem operations are protected by a single ReentrantReadWriteLock, which allows for having multiple concurrent readers to perform reads, and a single writer to perform writes. There are, however, operations whose execution has primarily reading nature, but occasionally they write. The finest example is processing block reports - currently the entire processing is done under writeLock(). With HDFS-395 (explicit deletion acks), processing a block report is primarily a read operation (reportDiff()) after which only very few blocks need to be updated. In fact, we noticed this number to be very low, or even zero blocks. It would be desirable to have an upgradeable read lock, which would allow for performing other reads during the first "read" part of reportDiff() (and possibly other operations. We implemented such mechanism, which provides writeLock(), readLock(), upgradeableReadLock, upgradeLock(), and downgradeLock(). I achieved this be emloying two ReentrantReadWriteLock's - one protects writes (lock1), the other one reads (lock2). Hence, we have: writeLock() lock1.writeLock().lock() lock2.writeLock().lock() readLock() lock2.readLock().lock() upgradeableReadLock() lock1.writeLock().lock() upgrade() lock2.writeLock().lock() -- Hence a writeLock() is essentially equivalent to upgradeableLock()+upgrade() - two writeLocks are mutually exclusive because of lock1.writeLock - a writeLock and upgradeableLock are mutually exclusive as above - readLock is mutually exclusive with upgradeableLock()+upgrade() OR writeLock because of lock2.writeLock - readLock() + writeLock() causes a deadlock, the same as currently - writeLock() + readLock() does not cause deadlocks -- I am conviced to the soundness of this mechanism. The overhead comes from having two locks, and in particular, writes need to acquire both of them. We deployed this feature, we used the upgradeableLock() ONLY for processing reports. Our initial, but not exhaustive experiments have shown that it had a very detrimental effect on the NN throughput - writes were taking up to twice as long. This is very unexpected, and hard to explain by only the overhead of acquiring additional lock for writes. I would like to ask for input, as maybe I am missing some fundamental problem here. I am attaching a java class which implements this locking mechanism. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133103#comment-13133103 ] Nathan Roberts commented on HDFS-2316: -- Is there a mechanism for versioning this API? Seems like we should probably have one. e.g. /webhdfs/1/ or /webhdfs/v1/ > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: WebHdfsAPI20111020.pdf > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[ https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-2450: -- Attachment: IP vs. Hostname.pdf Attaching a doc to clarify how this jira and 7510 interact. Will revise patch and repost per comments. > Only complete hostname is supported to access data via hdfs:// > -- > > Key: HDFS-2450 > URL: https://issues.apache.org/jira/browse/HDFS-2450 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Rajit Saha >Assignee: Daryn Sharp > Attachments: HDFS-2450-1.patch, HDFS-2450.patch, IP vs. Hostname.pdf > > > If my complete hostname is host1.abc.xyz.com, only complete hostname must be > used to access data via hdfs:// > I am running following in .20.205 Client to get data from .20.205 NN (host1) > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp > copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/ > copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: > hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/ > copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: > hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > Only following is supported > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133090#comment-13133090 ] Sanjay Radia commented on HDFS-2316: Alejandro raised the following 4 issues for discussion: # rename - should the target path contain the /webhdfs prefix since a client app will want to simply take the target path and use it as part of a read operation. # should getStatus return the paths with the /webhdfs prefix # Why is the scheme of the webhdfs file system "webhdfs:" and not "http:" # case sensitivity - make the parameters lower case rather than have the filter convert them since pathname and the user name should not be converted. My initial thoughts are: * for 1 and 2: the rename target path and the paths in the filestatus should NOT contain contain /webhdfs since /webhdfs is not really part of the parameters but a part of the rest api's "headers". * for scheme - i don't think we should use a prexisting scheme name. Nfs community has used webnfs as the scheme for accessing nfs over http. * I am fine with lower case parameters. > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: WebHdfsAPI20111020.pdf > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2488) Separate datatypes for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133075#comment-13133075 ] Hadoop QA commented on HDFS-2488: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500228/HDFS-2488.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.hdfs.TestSetrepIncreasing org.apache.hadoop.hdfs.server.datanode.TestInterDatanodeProtocol org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1409//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1409//console This message is automatically generated. > Separate datatypes for InterDatanodeProtocol > > > Key: HDFS-2488 > URL: https://issues.apache.org/jira/browse/HDFS-2488 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2488.txt > > > This jira separates for InterDatanodeProtocol the wire types from the types > used by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2489) Move commands Finalize and Register out of DatanodeCommand class.
[ https://issues.apache.org/jira/browse/HDFS-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2489: -- Status: Patch Available (was: Open) > Move commands Finalize and Register out of DatanodeCommand class. > - > > Key: HDFS-2489 > URL: https://issues.apache.org/jira/browse/HDFS-2489 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0, 0.24.0 > Environment: There are other subclasses available in separate files. > These commands should be moved to separate files as well. >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2489.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2489) Move commands Finalize and Register out of DatanodeCommand class.
[ https://issues.apache.org/jira/browse/HDFS-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2489: -- Attachment: HDFS-2489.patch > Move commands Finalize and Register out of DatanodeCommand class. > - > > Key: HDFS-2489 > URL: https://issues.apache.org/jira/browse/HDFS-2489 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0, 0.24.0 > Environment: There are other subclasses available in separate files. > These commands should be moved to separate files as well. >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2489.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2489) Move commands Finalize and Register out of DatanodeCommand class.
Move commands Finalize and Register out of DatanodeCommand class. - Key: HDFS-2489 URL: https://issues.apache.org/jira/browse/HDFS-2489 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0, 0.24.0 Environment: There are other subclasses available in separate files. These commands should be moved to separate files as well. Reporter: Suresh Srinivas Assignee: Suresh Srinivas -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2419) hadoop calls cat, tail, get, copyToLocal, on a secure cluster with an webhdfs uri fail with a 401
[ https://issues.apache.org/jira/browse/HDFS-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-2419. -- Resolution: Duplicate Duplicate HDFS-2416. > hadoop calls cat, tail, get, copyToLocal, on a secure cluster with an webhdfs > uri fail with a 401 > - > > Key: HDFS-2419 > URL: https://issues.apache.org/jira/browse/HDFS-2419 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Arpit Gupta >Assignee: Jitendra Nath Pandey > > a dfs -cat returns the following... > cat: Unauthorized (error code=401) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2416) distcp with a webhdfs uri on a secure cluster fails
[
https://issues.apache.org/jira/browse/HDFS-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133034#comment-13133034
]
Tsz Wo (Nicholas), SZE commented on HDFS-2416:
--
Thanks for fixing this. Some comments on the patch:
- For the code below,
-* typo: ivalid
-* Calling toString() is unnecessary.
{code}
+ throw new InvalidToken("token (" + identifier.toString()
+ + ") is ivalid, password doesn't match");
{code}
- The JspHelper.getUGI(request, conf) is not used except for test, how about
removing it?
- For DelegationTokenArgumentParam, let's have the class name TokenParam and
NAME="token". Then, it would be consistent with other parameters.
> distcp with a webhdfs uri on a secure cluster fails
> ---
>
> Key: HDFS-2416
> URL: https://issues.apache.org/jira/browse/HDFS-2416
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Jitendra Nath Pandey
> Attachments: HDFS-2416-branch-0.20-security.patch,
> HDFS-2419-branch-0.20-security.patch, HDFS-2419-branch-0.20-security.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2416) distcp with a webhdfs uri on a secure cluster fails
[ https://issues.apache.org/jira/browse/HDFS-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2416: - Issue Type: Sub-task (was: Bug) Parent: HDFS-2316 > distcp with a webhdfs uri on a secure cluster fails > --- > > Key: HDFS-2416 > URL: https://issues.apache.org/jira/browse/HDFS-2416 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.20.205.0 >Reporter: Arpit Gupta >Assignee: Jitendra Nath Pandey > Attachments: HDFS-2416-branch-0.20-security.patch, > HDFS-2419-branch-0.20-security.patch, HDFS-2419-branch-0.20-security.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133019#comment-13133019 ] Sanjay Radia commented on HDFS-2178: I believe it is bad for the prefix to configurable. We should use /webhdfs as the proxy as in webhdfs. That all URIs should be http://host:port/webhdfs/FilePath * It defeats the compatibility goal. Applications and tools should work transparently. * It prevents us from extending the proxy in the future. I believe we will be adding many more functions in the proxy and one cannot rely on adding a new port for each new function. * The configuration is another thing to get wrong and adds no value as far as i can see. > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.23.0 > > Attachments: HDFS-2178.patch, HDFSoverHTTP-API.html, HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133015#comment-13133015 ] Sanjay Radia commented on HDFS-2178: Good example that forced me to really think about compatibility between webhdfs and proxy. * I assume that the goal is : a customer writes an application that uses our rest APIs to read and write files AND that the application works in a transparent fashion for BOTH proxy or webhdfs. Do you agree with this? * 100-continue is a very good solution against the Http 1.1 spec in that the API works optimally with webhdfs and proxy: it allows the proxy to NOT redirect and the NN to redirect. Unfortunately it is not well supported by Jetty 6 (it is supported by Jetty 7, curl and httpclient). * Arpit's Put solution: ** Works really well with webhdfs with the limitation in jetty 6 and also in the future when jetty supports 100 continue correctly. ** However for proxy, the proxy ends up creating the file 2 times (The proxy should not redirect. Hence it is not an infinite loop.) * The getHandle solution has the unfortunate impact on the proxy that it always forces 2 operations when 1 operation would have been enough. It is also a solution that we will deprecate when jetty supports 100-continue. > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.23.0 > > Attachments: HDFS-2178.patch, HDFSoverHTTP-API.html, HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2480) Separate datatypes for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133014#comment-13133014 ] Hudson commented on HDFS-2480: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1146 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1146/]) HDFS-2480. Separate datatypes for NamenodeProtocol. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187505 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/BlockWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/BlocksWithLocationsWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/CheckpointSignatureWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/DatanodeInfoWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ExportedBlockKeysWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeProtocolServerSideTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeProtocolTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeWireProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamespaceInfoWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/RemoteEditLogManifestWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/RemoteEditLogWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/ExportedBlockKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckpointSignature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java > Separate datatypes for NamenodeProtocol > --- > > Key: HDFS-2480 > URL: https://issues.apache.org/jira/browse/HDFS-2480 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2480.txt, HDFS-2480.txt, HDFS-2480.txt > > > This jira separates for NamenodeProtocol the wire types from the types used > by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13133011#comment-13133011
]
Tsz Wo (Nicholas), SZE commented on HDFS-2485:
--
I believe the failed tests are not related to the patch.
> Improve code layout and constants in UnderReplicatedBlocks
> --
>
> Key: HDFS-2485
> URL: https://issues.apache.org/jira/browse/HDFS-2485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Attachments: HDFS-2485-improve-underreplicated.patch
>
> Original Estimate: 0.5h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Before starting HDFS-2472 I want to clean up the code in
> UnderReplicatedBlocks slightly
> # use constants for all the string levels
> # change the {{getUnderReplicatedBlockCount()}} method so that it works even
> if the corrupted block list is not the last queue
> # improve the javadocs
> # add some more curly braces and spaces to follow the style guidelines better
> This is a trivial change as behaviour will not change at all. If committed it
> will go into trunk and 0.23 so that patches between the two versions are easy
> to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-2485:
-
Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0)
Status: Open (was: Patch Available)
unexpected test failure, investigating
> Improve code layout and constants in UnderReplicatedBlocks
> --
>
> Key: HDFS-2485
> URL: https://issues.apache.org/jira/browse/HDFS-2485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Attachments: HDFS-2485-improve-underreplicated.patch
>
> Original Estimate: 0.5h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Before starting HDFS-2472 I want to clean up the code in
> UnderReplicatedBlocks slightly
> # use constants for all the string levels
> # change the {{getUnderReplicatedBlockCount()}} method so that it works even
> if the corrupted block list is not the last queue
> # improve the javadocs
> # add some more curly braces and spaces to follow the style guidelines better
> This is a trivial change as behaviour will not change at all. If committed it
> will go into trunk and 0.23 so that patches between the two versions are easy
> to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-2485 started by Steve Loughran.
> Improve code layout and constants in UnderReplicatedBlocks
> --
>
> Key: HDFS-2485
> URL: https://issues.apache.org/jira/browse/HDFS-2485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Attachments: HDFS-2485-improve-underreplicated.patch
>
> Original Estimate: 0.5h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Before starting HDFS-2472 I want to clean up the code in
> UnderReplicatedBlocks slightly
> # use constants for all the string levels
> # change the {{getUnderReplicatedBlockCount()}} method so that it works even
> if the corrupted block list is not the last queue
> # improve the javadocs
> # add some more curly braces and spaces to follow the style guidelines better
> This is a trivial change as behaviour will not change at all. If committed it
> will go into trunk and 0.23 so that patches between the two versions are easy
> to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2469) webhdfs create a file and send buffersize=0 hangs the request
[ https://issues.apache.org/jira/browse/HDFS-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-2469. -- Resolution: Duplicate Closing as a duplicate of HDFS-2427. > webhdfs create a file and send buffersize=0 hangs the request > - > > Key: HDFS-2469 > URL: https://issues.apache.org/jira/browse/HDFS-2469 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Arpit Gupta >Assignee: Tsz Wo (Nicholas), SZE > > Issue a create call and send buffersize=0 the user gets the redirect and upon > sending the request to the new location the request does not return with the > content. > Should we set a min and max value for buffersize? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132998#comment-13132998
]
Tsz Wo (Nicholas), SZE commented on HDFS-2485:
--
TestUnderReplicatedBlockQueues does not have license header. Otherwise, the
patch looks good.
> Improve code layout and constants in UnderReplicatedBlocks
> --
>
> Key: HDFS-2485
> URL: https://issues.apache.org/jira/browse/HDFS-2485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Attachments: HDFS-2485-improve-underreplicated.patch
>
> Original Estimate: 0.5h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Before starting HDFS-2472 I want to clean up the code in
> UnderReplicatedBlocks slightly
> # use constants for all the string levels
> # change the {{getUnderReplicatedBlockCount()}} method so that it works even
> if the corrupted block list is not the last queue
> # improve the javadocs
> # add some more curly braces and spaces to follow the style guidelines better
> This is a trivial change as behaviour will not change at all. If committed it
> will go into trunk and 0.23 so that patches between the two versions are easy
> to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2481) Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol
[
https://issues.apache.org/jira/browse/HDFS-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132979#comment-13132979
]
Hadoop QA commented on HDFS-2481:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12500216/testFailures2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.TestFileAppend2
org.apache.hadoop.hdfs.TestBalancerBandwidth
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1407//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1407//console
This message is automatically generated.
> Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol
>
>
> Key: HDFS-2481
> URL: https://issues.apache.org/jira/browse/HDFS-2481
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Sanjay Radia
> Attachments: testFailures1.patch, testFailures2.patch
>
>
> A few unit tests are failing, e.g.
> {noformat}
> Test set: org.apache.hadoop.hdfs.TestDistributedFileSystem
> ---
> Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.813 sec
> <<< FAILURE!
> testAllWithDualPort(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 0.254 sec <<< ERROR!
> java.io.IOException: java.io.IOException: Unknown protocol:
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:615)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1517)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2416) distcp with a webhdfs uri on a secure cluster fails
[ https://issues.apache.org/jira/browse/HDFS-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-2416: --- Attachment: HDFS-2416-branch-0.20-security.patch The problem happened because tasks connecting to the namenode fail in the absence of kerberos authentication. The fix is not to require kerberos authentication if token is present and instead authenticate the token. The patch fixes it by bypassing kerberos authentication in AuthFilter in hdfs if token is there and verifies the token in JspHelper. The same code path is used for token authentication in hftp also. > distcp with a webhdfs uri on a secure cluster fails > --- > > Key: HDFS-2416 > URL: https://issues.apache.org/jira/browse/HDFS-2416 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Arpit Gupta >Assignee: Jitendra Nath Pandey > Attachments: HDFS-2416-branch-0.20-security.patch, > HDFS-2419-branch-0.20-security.patch, HDFS-2419-branch-0.20-security.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2480) Separate datatypes for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132953#comment-13132953 ] Hudson commented on HDFS-2480: -- Integrated in Hadoop-Hdfs-trunk-Commit #1209 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1209/]) HDFS-2480. Separate datatypes for NamenodeProtocol. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187505 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/BlockWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/BlocksWithLocationsWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/CheckpointSignatureWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/DatanodeInfoWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ExportedBlockKeysWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeProtocolServerSideTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeProtocolTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeWireProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamespaceInfoWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/RemoteEditLogManifestWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/RemoteEditLogWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/ExportedBlockKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckpointSignature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java > Separate datatypes for NamenodeProtocol > --- > > Key: HDFS-2480 > URL: https://issues.apache.org/jira/browse/HDFS-2480 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2480.txt, HDFS-2480.txt, HDFS-2480.txt > > > This jira separates for NamenodeProtocol the wire types from the types used > by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2488) Separate datatypes for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2488: -- Attachment: HDFS-2488.txt > Separate datatypes for InterDatanodeProtocol > > > Key: HDFS-2488 > URL: https://issues.apache.org/jira/browse/HDFS-2488 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2488.txt > > > This jira separates for InterDatanodeProtocol the wire types from the types > used by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2488) Separate datatypes for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2488: -- Attachment: (was: HDFS-2488.txt) > Separate datatypes for InterDatanodeProtocol > > > Key: HDFS-2488 > URL: https://issues.apache.org/jira/browse/HDFS-2488 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > > This jira separates for InterDatanodeProtocol the wire types from the types > used by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2480) Separate datatypes for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132951#comment-13132951 ] Hudson commented on HDFS-2480: -- Integrated in Hadoop-Common-trunk-Commit #1131 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1131/]) HDFS-2480. Separate datatypes for NamenodeProtocol. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187505 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/BlockWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/BlocksWithLocationsWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/CheckpointSignatureWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/DatanodeInfoWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/ExportedBlockKeysWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeProtocolServerSideTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeProtocolTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeWireProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamespaceInfoWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/RemoteEditLogManifestWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/RemoteEditLogWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/ExportedBlockKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CheckpointSignature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeProtocol.java > Separate datatypes for NamenodeProtocol > --- > > Key: HDFS-2480 > URL: https://issues.apache.org/jira/browse/HDFS-2480 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2480.txt, HDFS-2480.txt, HDFS-2480.txt > > > This jira separates for NamenodeProtocol the wire types from the types used > by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132949#comment-13132949 ] Konstantin Shvachko commented on HDFS-2452: --- Let's commit the fix and file another jira for the test, dependent on the mavenization progress. > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2488) Separate datatypes for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132939#comment-13132939 ] Hadoop QA commented on HDFS-2488: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500221/HDFS-2488.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1408//console This message is automatically generated. > Separate datatypes for InterDatanodeProtocol > > > Key: HDFS-2488 > URL: https://issues.apache.org/jira/browse/HDFS-2488 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2488.txt > > > This jira separates for InterDatanodeProtocol the wire types from the types > used by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132941#comment-13132941 ] Konstantin Boudnik commented on HDFS-2452: -- Right. However, patch for trunk will depends on completion of Mavenization work. Right now fault-injection is _completely_ unsupported by trunk build system. Pity ;( > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132937#comment-13132937 ] Hadoop QA commented on HDFS-1580: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12500207/HDFS-1580.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.TestAbandonBlock org.apache.hadoop.hdfs.server.namenode.TestBackupNode org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.TestRestartDFS +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1406//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1406//console This message is automatically generated. > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2480) Separate datatypes for NamenodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2480: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch. Javac warning as I commented earlier is due to use of deprecated methods. No tests are added as the protocol related changes are tested using existing tests. > Separate datatypes for NamenodeProtocol > --- > > Key: HDFS-2480 > URL: https://issues.apache.org/jira/browse/HDFS-2480 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2480.txt, HDFS-2480.txt, HDFS-2480.txt > > > This jira separates for NamenodeProtocol the wire types from the types used > by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2488) Separate datatypes for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2488: -- Status: Patch Available (was: Open) > Separate datatypes for InterDatanodeProtocol > > > Key: HDFS-2488 > URL: https://issues.apache.org/jira/browse/HDFS-2488 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2488.txt > > > This jira separates for InterDatanodeProtocol the wire types from the types > used by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2488) Separate datatypes for InterDatanodeProtocol
Separate datatypes for InterDatanodeProtocol Key: HDFS-2488 URL: https://issues.apache.org/jira/browse/HDFS-2488 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.0, 0.24.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-2488.txt This jira separates for InterDatanodeProtocol the wire types from the types used by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2488) Separate datatypes for InterDatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2488: -- Attachment: HDFS-2488.txt > Separate datatypes for InterDatanodeProtocol > > > Key: HDFS-2488 > URL: https://issues.apache.org/jira/browse/HDFS-2488 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2488.txt > > > This jira separates for InterDatanodeProtocol the wire types from the types > used by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2410) Further clean up hard-coded configuration keys
[ https://issues.apache.org/jira/browse/HDFS-2410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132900#comment-13132900 ] Jitendra Nath Pandey commented on HDFS-2410: The changes to JsonUtil don't seem to belong here. Please rebase the patch against latest trunk. The changes to conf keys look good. > Further clean up hard-coded configuration keys > -- > > Key: HDFS-2410 > URL: https://issues.apache.org/jira/browse/HDFS-2410 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node, test >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Minor > Attachments: HDFS-2410.txt > > > HDFS code is littered with hardcoded config key names. This jira changes to > use DFSConfigKeys constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2487) HA: Administrative CLI to control HA daemons
HA: Administrative CLI to control HA daemons Key: HDFS-2487 URL: https://issues.apache.org/jira/browse/HDFS-2487 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs client Affects Versions: HA branch (HDFS-1623) Reporter: Aaron T. Myers Assignee: Aaron T. Myers We'll need to have some way of controlling the HA nodes while they're live, probably by adding some more commands to dfsadmin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2481) Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol
[
https://issues.apache.org/jira/browse/HDFS-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Radia updated HDFS-2481:
---
Status: Patch Available (was: Open)
> Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol
>
>
> Key: HDFS-2481
> URL: https://issues.apache.org/jira/browse/HDFS-2481
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Sanjay Radia
> Attachments: testFailures1.patch, testFailures2.patch
>
>
> A few unit tests are failing, e.g.
> {noformat}
> Test set: org.apache.hadoop.hdfs.TestDistributedFileSystem
> ---
> Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.813 sec
> <<< FAILURE!
> testAllWithDualPort(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 0.254 sec <<< ERROR!
> java.io.IOException: java.io.IOException: Unknown protocol:
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:615)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1517)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2481) Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol
[
https://issues.apache.org/jira/browse/HDFS-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Radia updated HDFS-2481:
---
Attachment: testFailures2.patch
> Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol
>
>
> Key: HDFS-2481
> URL: https://issues.apache.org/jira/browse/HDFS-2481
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Sanjay Radia
> Attachments: testFailures1.patch, testFailures2.patch
>
>
> A few unit tests are failing, e.g.
> {noformat}
> Test set: org.apache.hadoop.hdfs.TestDistributedFileSystem
> ---
> Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.813 sec
> <<< FAILURE!
> testAllWithDualPort(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 0.254 sec <<< ERROR!
> java.io.IOException: java.io.IOException: Unknown protocol:
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:615)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1517)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2481) Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol
[
https://issues.apache.org/jira/browse/HDFS-2481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Radia updated HDFS-2481:
---
Status: Open (was: Patch Available)
> Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol
>
>
> Key: HDFS-2481
> URL: https://issues.apache.org/jira/browse/HDFS-2481
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Sanjay Radia
> Attachments: testFailures1.patch, testFailures2.patch
>
>
> A few unit tests are failing, e.g.
> {noformat}
> Test set: org.apache.hadoop.hdfs.TestDistributedFileSystem
> ---
> Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.813 sec
> <<< FAILURE!
> testAllWithDualPort(org.apache.hadoop.hdfs.TestDistributedFileSystem) Time
> elapsed: 0.254 sec <<< ERROR!
> java.io.IOException: java.io.IOException: Unknown protocol:
> org.apache.hadoop.hdfs.protocol.ClientProtocol
> at
> org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:615)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1517)
> ...
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[
https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132866#comment-13132866
]
Suresh Srinivas commented on HDFS-2450:
---
# FileSystem.java
#* Please cache canonical URI for defaultURI and FileSystem URI, instead of
getting it call to checkPath.
#* Please add more details to getCanonicalUri() method as to what is expected
in uri's hostname (FQDN).
#* Code duplication in the following snippet can be avoided by creating a
protected static method.
{noformat}
+if (thisScheme.equalsIgnoreCase(defaultUri.getScheme())) {
+ if (thisAuthority.equalsIgnoreCase(defaultUri.getAuthority()))
+return;
+ defaultUri = NetUtils.getCanonicalUri(defaultUri, getDefaultPort());
+ thisAuthority = getCanonicalUri().getAuthority();
+ if (thisAuthority.equalsIgnoreCase(defaultUri.getAuthority()))
+return;
{noformat}
# DistributedFileSystem.java - The existing methods checkPath() and
makeQualified() allowed default ports exclusively (indicated in the javadoc). I
am not sure why that was done. Why is this patch removing that?
# In tests how does host become host.a.b?
> Only complete hostname is supported to access data via hdfs://
> --
>
> Key: HDFS-2450
> URL: https://issues.apache.org/jira/browse/HDFS-2450
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.20.205.0
>Reporter: Rajit Saha
>Assignee: Daryn Sharp
> Attachments: HDFS-2450-1.patch, HDFS-2450.patch
>
>
> If my complete hostname is host1.abc.xyz.com, only complete hostname must be
> used to access data via hdfs://
> I am running following in .20.205 Client to get data from .20.205 NN (host1)
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp
> copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal ... ]
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
> hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal ... ]
> $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
> hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal ... ]
> Only following is supported
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132864#comment-13132864
]
Hadoop QA commented on HDFS-2485:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12500201/HDFS-2485-improve-underreplicated.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.TestFileAppend2
org.apache.hadoop.hdfs.TestBalancerBandwidth
org.apache.hadoop.hdfs.TestRestartDFS
org.apache.hadoop.hdfs.TestDistributedFileSystem
org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1405//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1405//console
This message is automatically generated.
> Improve code layout and constants in UnderReplicatedBlocks
> --
>
> Key: HDFS-2485
> URL: https://issues.apache.org/jira/browse/HDFS-2485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Attachments: HDFS-2485-improve-underreplicated.patch
>
> Original Estimate: 0.5h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Before starting HDFS-2472 I want to clean up the code in
> UnderReplicatedBlocks slightly
> # use constants for all the string levels
> # change the {{getUnderReplicatedBlockCount()}} method so that it works even
> if the corrupted block list is not the last queue
> # improve the javadocs
> # add some more curly braces and spaces to follow the style guidelines better
> This is a trivial change as behaviour will not change at all. If committed it
> will go into trunk and 0.23 so that patches between the two versions are easy
> to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132836#comment-13132836 ] Konstantin Shvachko commented on HDFS-2452: --- We should make a patch for trunk too. > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1580: - Attachment: HDFS-1580.diff This is the final piece to allow the loading of custom implementations of JournalManager. There is another change HDFS-2334 which adds closeable to JournalManager, but that may not be absolutely necessary for all journal types. (it is for bookkeeper) There's 2 changes: 1) I've changes the interfaces(JournalManager, EditLogInputStream & EditLogOutputStream) so that they can be implemented outside of the org.apache.hadoop.hdfs.server.namenode. 2) Pluggable creation of journal managers. When FSEditLog is creating JournalManagers from dfs.namenode.edits.dir, and it encounters a URI with a schema different to "file" it loads the name of the implementing class from "dfs.namenode.edits.journal-plugin.". This class must implement JournalManager and have a constructor which takes (Configuration, URI). > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1580: - Fix Version/s: (was: 0.23.0) 0.24.0 HA branch (HDFS-1623) Status: Patch Available (was: Open) > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly >Assignee: Jitendra Nath Pandey > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, > EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, > HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, > generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-2485:
-
Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0)
Status: Patch Available (was: In Progress)
> Improve code layout and constants in UnderReplicatedBlocks
> --
>
> Key: HDFS-2485
> URL: https://issues.apache.org/jira/browse/HDFS-2485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Attachments: HDFS-2485-improve-underreplicated.patch
>
> Original Estimate: 0.5h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Before starting HDFS-2472 I want to clean up the code in
> UnderReplicatedBlocks slightly
> # use constants for all the string levels
> # change the {{getUnderReplicatedBlockCount()}} method so that it works even
> if the corrupted block list is not the last queue
> # improve the javadocs
> # add some more curly braces and spaces to follow the style guidelines better
> This is a trivial change as behaviour will not change at all. If committed it
> will go into trunk and 0.23 so that patches between the two versions are easy
> to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-2485:
-
Attachment: HDFS-2485-improve-underreplicated.patch
This code adds constants and other cleanup, and adds the unit tests to verify
that the current behaviour is as expected.
> Improve code layout and constants in UnderReplicatedBlocks
> --
>
> Key: HDFS-2485
> URL: https://issues.apache.org/jira/browse/HDFS-2485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Attachments: HDFS-2485-improve-underreplicated.patch
>
> Original Estimate: 0.5h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Before starting HDFS-2472 I want to clean up the code in
> UnderReplicatedBlocks slightly
> # use constants for all the string levels
> # change the {{getUnderReplicatedBlockCount()}} method so that it works even
> if the corrupted block list is not the last queue
> # improve the javadocs
> # add some more curly braces and spaces to follow the style guidelines better
> This is a trivial change as behaviour will not change at all. If committed it
> will go into trunk and 0.23 so that patches between the two versions are easy
> to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work logged] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel\#worklog-{worklog.getId()}
]
Steve Loughran logged work on HDFS-2485:
Author: Steve Loughran
Created on: 21/Oct/11 16:27
Start Date: 21/Oct/11 16:27
Worklog Time Spent: 1h
Work Description: code plus tests
Issue Time Tracking
---
Worklog Id: (was: 12334)
Time Spent: 1h
Remaining Estimate: 0h (was: 0.5h)
> Improve code layout and constants in UnderReplicatedBlocks
> --
>
> Key: HDFS-2485
> URL: https://issues.apache.org/jira/browse/HDFS-2485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Attachments: HDFS-2485-improve-underreplicated.patch
>
> Original Estimate: 0.5h
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Before starting HDFS-2472 I want to clean up the code in
> UnderReplicatedBlocks slightly
> # use constants for all the string levels
> # change the {{getUnderReplicatedBlockCount()}} method so that it works even
> if the corrupted block list is not the last queue
> # improve the javadocs
> # add some more curly braces and spaces to follow the style guidelines better
> This is a trivial change as behaviour will not change at all. If committed it
> will go into trunk and 0.23 so that patches between the two versions are easy
> to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[
https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132785#comment-13132785
]
Konstantin Boudnik commented on HDFS-2452:
--
Hi Uma.
Aspect either work on not :) - they are just a pieces of byte code physically
inserted into the target class. The reason you see {{DataXceiver.java:36}} in
the stack is because there's no source code line anymore to tie with.
An easy way to find out what's going on is to disassemble the class file. When
you do it you see this:
{code}
59 DataNode getDataNode()
60 {
61 return datanode;
62 }
63
64 public void run()
65 {
66 run_aroundBody1$advice(this, DataXceiverAspects.aspectOf(), this,
null);
67 }
68
69 protected void opReadBlock(DataInputStream in, Block block, long
startOffset, long length, String clientName,
70 Token blockToken)
71 throws IOException
{code}
I'd say it should work as expected. If an unexpected behavior is seen it is
likely that we instrumented it on somewhat incorrect assumptions.
> OutOfMemoryError in DataXceiverServer takes down the DataNode
> -
>
> Key: HDFS-2452
> URL: https://issues.apache.org/jira/browse/HDFS-2452
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.22.0
>Reporter: Konstantin Shvachko
>Assignee: Uma Maheswara Rao G
> Fix For: 0.22.0
>
> Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch,
> HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch,
> HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch,
> HDFS-2452-22branch_with-around_.patch
>
>
> OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn
> a new data transfer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132761#comment-13132761 ] Alejandro Abdelnur commented on HDFS-2178: -- Arpit, I'm referring to the hdfsproxy usecase where the redirection goes back to itself. > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.23.0 > > Attachments: HDFS-2178.patch, HDFSoverHTTP-API.html, HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[
https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132752#comment-13132752
]
Arpit Gupta commented on HDFS-2178:
---
@Alenjandro
{quote}
100 Continue issue, the problem I see with Arpit's solution is that if somebody
is creating/appending an empty stream it will go into an infinite loop because
the server will not see any content and it will respond with a 307 & Location
over and over. IMO we need a getCreateHandle & getAppendHandle calls.
{quote}
I am not quite sure why one would go to an infinite loop. When i make the
initial call i do not send data and get the expected 307 and make the put call
to the datanode with the data. Regardless i tested the case where on the
redirect i sent empty content and webhdfs was able to create an empty file
without any issues.
> Contributing Hoop to HDFS, replacement for HDFS proxy with read/write
> capabilities
> --
>
> Key: HDFS-2178
> URL: https://issues.apache.org/jira/browse/HDFS-2178
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 0.23.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Fix For: 0.23.0
>
> Attachments: HDFS-2178.patch, HDFSoverHTTP-API.html, HdfsHttpAPI.pdf
>
>
> We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement)
> for HDFS Proxy.
> Hoop provides access to all Hadoop Distributed File System (HDFS) operations
> (read and write) over HTTP/S.
> The Hoop server component is a REST HTTP gateway to HDFS supporting all file
> system operations. It can be accessed using standard HTTP tools (i.e. curl
> and wget), HTTP libraries from different programing languages (i.e. Perl,
> Java Script) as well as using the Hoop client. The Hoop server component is a
> standard Java web-application and it has been implemented using Jersey
> (JAX-RS).
> The Hoop client component is an implementation of Hadoop FileSystem client
> that allows using the familiar Hadoop filesystem API to access HDFS data
> through a Hoop server.
> Repo: https://github.com/cloudera/hoop
> Docs: http://cloudera.github.com/hoop
> Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/
> Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for
> Kerberos HTTP SPNEGO authentication).
> To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be
> done first, as well as the Alfredo contribution (HADOOP-7119).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2486) Review issues with UnderReplicatedBlocks
Review issues with UnderReplicatedBlocks Key: HDFS-2486 URL: https://issues.apache.org/jira/browse/HDFS-2486 Project: Hadoop HDFS Issue Type: Task Components: name-node Affects Versions: 0.23.0 Reporter: Steve Loughran Priority: Minor Here are some things I've noted in the UnderReplicatedBlocks class that someone else should review and consider if the code is correct. If not, they are easy to fix. remove(Block block, int priLevel) is not synchronized, and as the inner classes are not, there is a risk of race conditions there. some of the code assumes that getPriority can return the value LEVEL, and if so does not attempt to queue the blocks. As this return value is not currently possible, those checks can be removed. The queue gives priority to blocks whose replication count is less than a third of its expected count over those that are "normally under replicated". While this is good for ensuring that files scheduled for large replication are replicated fast, it may not be the best strategy for maintaining data integrity. For that it may be better to give whichever blocks have only two replicas priority over blocks that may, for example, already have 3 out of 10 copies in the filesystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132732#comment-13132732 ] Nathan Roberts commented on HDFS-2316: -- Hi Nicholas, some quick comments from first read: * ":" and "http://:" seem to be used interchangeably. We should be consistent where possible. * Why doesn't "curl -i -L "http://:/webhdfs/" just work? Do we really need to specify op=OPEN for this very simple, common case? * I believe "http://:" should be "http://:" in append. * Need format of responses spelled out. * It would be nice if we could document the possible error responses as well. * Since a single datanode will be performing the write of a potentially large file, does that mean that file will have an entire copy on that node (due to block placement strategies)? That doesn't seem desirable.. * Is a SHORT sufficient for buffersize? * Do we need a renewlease? How will very slow writers be handled? * Once I have file block locations, can I go directly to those datanodes to retrieve rather than using content_range and always following a redirect? * Do we need flush/sync? > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: WebHdfsAPI20111020.pdf > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2012) Recurring failure of TestBalancer due to incorrect treatment of nodes whose utilization equals avgUtilization.
[
https://issues.apache.org/jira/browse/HDFS-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132731#comment-13132731
]
Uma Maheswara Rao G commented on HDFS-2012:
---
We need to handle one more condition i think,
{CODE}
if (getUtilization(datanode) > avgUtilization) {
datanodeS = new Source(datanode, avgUtilization, threshold);
if (isAboveAvgUtilized(datanodeS)) {
{CODE}
we handled only in isAboveAvgUtilized method to add the DN when dn utilization
is equal to avgUtilization.
But that check is happening before as well.
> Recurring failure of TestBalancer due to incorrect treatment of nodes whose
> utilization equals avgUtilization.
> --
>
> Key: HDFS-2012
> URL: https://issues.apache.org/jira/browse/HDFS-2012
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer, test
>Affects Versions: 0.22.0
>Reporter: Aaron T. Myers
>Assignee: Uma Maheswara Rao G
>Priority: Blocker
> Fix For: 0.22.0, 0.24.0
>
> Attachments: HDFS-2012-0.22Branch.patch, HDFS-2012-Trunk.patch,
> HDFS-2012.patch, TestBalancerLog.html
>
>
> This has been failing on Hudson for the last two builds and fails on my local
> box as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132717#comment-13132717 ] Uma Maheswara Rao G commented on HDFS-2452: --- Hi Cos, Looks that aspects not working reliably.Still it is throwing the error after spawning the DataXceiver thread. So, parent thread (DataXceiverServer) is not getting that exception. [junit] at java.lang.Thread.run(Thread.java:662) [junit] Exception in thread "org.apache.hadoop.hdfs.server.datanode.DataXceiver@6e2335" java.lang.OutOfMemoryError: Pretend there's no more memory [junit] at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run_aroundBody1$advice(DataXceiver.java:36) [junit] at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:1) [junit] at java.lang.Thread.run(Thread.java:662) [junit] Exception in thread "org.apache.hadoop.hdfs.server.datanode.DataXceiver@576165" java.lang.OutOfMemoryError: Pretend there's no more memory [junit] at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run_aroundBody1$advice(DataXceiver.java:36) [junit] at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:1) > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132692#comment-13132692 ] Alejandro Abdelnur commented on HDFS-2316: -- --- @Nicholas, Thanks for the API document. In general it looks OK. A few comments: * GET GETHOMEDIRECTORY operation is missing. * The GETFILEBLOCKLOCATIONS, GETDELEGATIONTOKEN, RENEWDELEGATIONTOKEN, CANCELDELEGATIONTOKEN operations seem to be the ones that don't make sense (at the moment) in a proxy scenario. We should make those operations as optional. * The 'doas' query parameter is missing, this is required to enable proxyuser functionality. * The 'user.name' query parameter is optional as this is used only in the case of pseudo authentication, in the case of other authentication mechanism the username will be taken for the authentication credentials. * The document does not define any of the JSON responses nor error codes and JSON error messages. I assume you are taking the JSON responses in the doc posted in HDFS-2178. Still this has to be augmented for checksum and content-summary responses. * IMO we need operations to get create and append handles, reason in my response to Sanjay in HDFS-2178 (https://issues.apache.org/jira/browse/HDFS-2178?focusedCommentId=13132691&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13132691) * The webhdfs prefix should be optional/configurable and it should be provided by server on a 'filsystem.get' operation. > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: WebHdfsAPI20111020.pdf > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132691#comment-13132691 ] Alejandro Abdelnur commented on HDFS-2178: -- --- @Sanjay, * *100 Continue issue*, the problem I see with Arpit's solution is that if somebody is creating/appending an empty stream it will go into an infinite loop because the server will not see any content and it will respond with a 307 & Location over and over. IMO we need a getCreateHandle & getAppendHandle calls. * *Proxy and webhdfs API the same or almost the same*, for the operations that do not make sense in one an 'unsupported operation' should be returned, for the operations that are supported in both, the call should be the same. This would allow applications built against proxy to work with webhdfs and viceversa (assuming they account for the unsupported operations in each case). Note that redirection in proxy makes sense to ensure authentication, you don't want to put/post (create/append) data just to find the request being rejected because of not authentication credentials. * *API*, see comment @Nicholas in HDFS-2316. * *Pure proxy vs hdfs proxy*. I'm not understanding the question 'Would it make sense ...' > Contributing Hoop to HDFS, replacement for HDFS proxy with read/write > capabilities > -- > > Key: HDFS-2178 > URL: https://issues.apache.org/jira/browse/HDFS-2178 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 0.23.0 > > Attachments: HDFS-2178.patch, HDFSoverHTTP-API.html, HdfsHttpAPI.pdf > > > We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) > for HDFS Proxy. > Hoop provides access to all Hadoop Distributed File System (HDFS) operations > (read and write) over HTTP/S. > The Hoop server component is a REST HTTP gateway to HDFS supporting all file > system operations. It can be accessed using standard HTTP tools (i.e. curl > and wget), HTTP libraries from different programing languages (i.e. Perl, > Java Script) as well as using the Hoop client. The Hoop server component is a > standard Java web-application and it has been implemented using Jersey > (JAX-RS). > The Hoop client component is an implementation of Hadoop FileSystem client > that allows using the familiar Hadoop filesystem API to access HDFS data > through a Hoop server. > Repo: https://github.com/cloudera/hoop > Docs: http://cloudera.github.com/hoop > Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ > Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for > Kerberos HTTP SPNEGO authentication). > To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be > done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
Improve code layout and constants in UnderReplicatedBlocks
--
Key: HDFS-2485
URL: https://issues.apache.org/jira/browse/HDFS-2485
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Affects Versions: 0.23.0, 0.24.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Trivial
Before starting HDFS-2472 I want to clean up the code in UnderReplicatedBlocks
slightly
# use constants for all the string levels
# change the {{getUnderReplicatedBlockCount()}} method so that it works even if
the corrupted block list is not the last queue
# improve the javadocs
# add some more curly braces and spaces to follow the style guidelines better
This is a trivial change as behaviour will not change at all. If committed it
will go into trunk and 0.23 so that patches between the two versions are easy
to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HDFS-2485) Improve code layout and constants in UnderReplicatedBlocks
[
https://issues.apache.org/jira/browse/HDFS-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-2485 started by Steve Loughran.
> Improve code layout and constants in UnderReplicatedBlocks
> --
>
> Key: HDFS-2485
> URL: https://issues.apache.org/jira/browse/HDFS-2485
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Trivial
> Original Estimate: 0.5h
> Remaining Estimate: 0.5h
>
> Before starting HDFS-2472 I want to clean up the code in
> UnderReplicatedBlocks slightly
> # use constants for all the string levels
> # change the {{getUnderReplicatedBlockCount()}} method so that it works even
> if the corrupted block list is not the last queue
> # improve the javadocs
> # add some more curly braces and spaces to follow the style guidelines better
> This is a trivial change as behaviour will not change at all. If committed it
> will go into trunk and 0.23 so that patches between the two versions are easy
> to apply
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132673#comment-13132673 ] Hudson commented on HDFS-2452: -- Integrated in Hadoop-Hdfs-22-branch #101 (See [https://builds.apache.org/job/Hadoop-Hdfs-22-branch/101/]) HDFS-2452. OutOfMemoryError in DataXceiverServer takes down the DataNode. Contributed by Uma Maheswara Rao. cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187168 Files : * /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java * /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java * /hadoop/common/branches/branch-0.22/hdfs/src/test/aop/org/apache/hadoop/hdfs/server/datanode/DataXceiverAspects.aj * /hadoop/common/branches/branch-0.22/hdfs/src/test/aop/org/apache/hadoop/hdfs/server/datanode/TestFiDataXceiverServer.java > OutOfMemoryError in DataXceiverServer takes down the DataNode > - > > Key: HDFS-2452 > URL: https://issues.apache.org/jira/browse/HDFS-2452 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2452-22Branch.2.patch, HDFS-2452-22branch.1.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch, > HDFS-2452-22branch.patch, HDFS-2452-22branch_with-around_.patch, > HDFS-2452-22branch_with-around_.patch > > > OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn > a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HDFS-2472) Extend UnderReplicatedBlocks queue to give blocks whose existing copies are all on a single rack priority over multi-rack blocks
[ https://issues.apache.org/jira/browse/HDFS-2472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-2472 started by Steve Loughran. > Extend UnderReplicatedBlocks queue to give blocks whose existing copies are > all on a single rack priority over multi-rack blocks > > > Key: HDFS-2472 > URL: https://issues.apache.org/jira/browse/HDFS-2472 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > > Google's Availability in Globally Distributed Storage Systems paper > [http://research.google.com/pubs/archive/36737.pdf] shows that storage node > failures are often correlated with other failures on the same rack. This > could be due to rack-local failures: switches, power, etc, or operations > actions (take down a whole rack for a rolling OS upgrade). Whatever the root > cause, the paper argues (section 5.2) that rack-aware placement would > increase the MTTF of a single block by a factor of three (typically). Some > decisions can be made a block placement time, but that would be a separate > issue. > Here I propose giving priority to blocks that are under replicated and where > all blocks are on the same rack above those blocks that are under-replicated > and the remaining blocks are on different racks. > # Provided the failure does not take down the entire rack in one go (e.g. > switch failure), this policy would decrease the time in which all blocks > would be on a single rack, so reduce the consequences of a rack failure. > # On a single-rack system, all under-replicated blocks would go into this > queue, so the state would effectively be that of today's system. > This may make the demand for off-rack bandwidth ramp up immediately, because > priority will be given to blocks that must be replicated off rack. I am not > sure that it will, however, as multi-rack replication policies would generate > the same amount of traffic to re-replicate the blocks anyway. > The main barrier to implementing this feature is that currently the > UnderReplicatedBlocks queues do not get provided with any information about > block locality. We'd need to change the add() method to take information > about where the current replicas are, and the BlockManager would have to > provide some information as to whether the block was rack-local, not-rack > local or unknown; the latter because the PendingReplicationBlocks structure > does not know where things come from. "unknown" items would have to be given > the same priority as rack-only blocks (pessimistic) or the same priority as > multi-rack blocks (optimistic). I would be biased towards the pessimistic > approach as it would ensure that on single-rack systems there would be no > obvious change in behaviour. On a multi-rack system it would give priority to > timed out PendingReplication blocks ahead of multi-rack under-replicated > blocks. That may be a good thing in itself -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2459) Separate datatypes for Journal protocol
[ https://issues.apache.org/jira/browse/HDFS-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132659#comment-13132659 ] Hudson commented on HDFS-2459: -- Integrated in Hadoop-Hdfs-trunk #837 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/837/]) HDFS-2459. Separate datatypes for Journal Protocol. (suresh) suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186896 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/JournalProtocolServerSideTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/JournalProtocolTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/JournalWireProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeRegistrationWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/StorageInfoWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/JournalProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeRegistration.java > Separate datatypes for Journal protocol > --- > > Key: HDFS-2459 > URL: https://issues.apache.org/jira/browse/HDFS-2459 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2459.txt, HDFS-2459.txt, HDFS-2459.txt > > > This jira separates for JournalProtocol the wire types from the types used by > the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2427) webhdfs mkdirs api call creates path with 777 permission, we should default it to 755
[ https://issues.apache.org/jira/browse/HDFS-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132658#comment-13132658 ] Hudson commented on HDFS-2427: -- Integrated in Hadoop-Hdfs-trunk #837 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/837/]) HDFS-2427. Change the default permission in webhdfs to 755 and add range check/validation for all parameters. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187140 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/AccessTimeParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BlockSizeParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BufferSizeParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/DestinationParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/DstPathParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/IntegerParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/LengthParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/LongParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ModificationTimeParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/OffsetParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/PermissionParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ReplicationParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ShortParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources/TestParam.java > webhdfs mkdirs api call creates path with 777 permission, we should default > it to 755 > - > > Key: HDFS-2427 > URL: https://issues.apache.org/jira/browse/HDFS-2427 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.20.205.0 >Reporter: Arpit Gupta >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.20.205.1, 0.20.206.0, 0.24.0 > > Attachments: h2427_20111019.patch, h2427_20111019b.patch, > h2427_20111020.patch, h2427_20111020_svn_mv.patch, > h2427_20111020_svn_mv_0.20s.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2459) Separate datatypes for Journal protocol
[ https://issues.apache.org/jira/browse/HDFS-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132646#comment-13132646 ] Hudson commented on HDFS-2459: -- Integrated in Hadoop-Mapreduce-trunk #867 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/867/]) HDFS-2459. Separate datatypes for Journal Protocol. (suresh) suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186896 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/JournalProtocolServerSideTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/JournalProtocolTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/JournalWireProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/NamenodeRegistrationWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/StorageInfoWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/JournalProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/NamenodeRegistration.java > Separate datatypes for Journal protocol > --- > > Key: HDFS-2459 > URL: https://issues.apache.org/jira/browse/HDFS-2459 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2459.txt, HDFS-2459.txt, HDFS-2459.txt > > > This jira separates for JournalProtocol the wire types from the types used by > the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2427) webhdfs mkdirs api call creates path with 777 permission, we should default it to 755
[ https://issues.apache.org/jira/browse/HDFS-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132645#comment-13132645 ] Hudson commented on HDFS-2427: -- Integrated in Hadoop-Mapreduce-trunk #867 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/867/]) HDFS-2427. Change the default permission in webhdfs to 755 and add range check/validation for all parameters. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1187140 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/AccessTimeParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BlockSizeParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/BufferSizeParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/DestinationParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/DstPathParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/IntegerParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/LengthParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/LongParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ModificationTimeParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/OffsetParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/PermissionParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ReplicationParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/ShortParam.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/resources/TestParam.java > webhdfs mkdirs api call creates path with 777 permission, we should default > it to 755 > - > > Key: HDFS-2427 > URL: https://issues.apache.org/jira/browse/HDFS-2427 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.20.205.0 >Reporter: Arpit Gupta >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.20.205.1, 0.20.206.0, 0.24.0 > > Attachments: h2427_20111019.patch, h2427_20111019b.patch, > h2427_20111020.patch, h2427_20111020_svn_mv.patch, > h2427_20111020_svn_mv_0.20s.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2484) checkLease should throw FileNotFoundException when file does not exist
[
https://issues.apache.org/jira/browse/HDFS-2484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132498#comment-13132498
]
Konstantin Shvachko commented on HDFS-2484:
---
The exception is like this:
{code}
2011-10-21 00:17:52,610 WARN org.apache.hadoop.fs.slive.CreateOp: Error with
creating
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on
/test/slive/slive/data/sl_dir_26/sl_file_5 File does not exist. Holder
DFSClient_attempt_201110141824_0140_m_000164_0 does not have any open files.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1729)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1720)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1775)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1758)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.completeInternal(NameNode.java:953)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:942)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:349)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1482)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1028)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
at $Proxy1.complete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:84)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy1.complete(Unknown Source)
at
org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:1518)
at
org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:1505)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:66)
at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:91)
at org.apache.hadoop.fs.slive.CreateOp.run(CreateOp.java:152)
at org.apache.hadoop.fs.slive.ObserveableOp.run(ObserveableOp.java:63)
at
org.apache.hadoop.fs.slive.SliveMapper.runOperation(SliveMapper.java:136)
at org.apache.hadoop.fs.slive.SliveMapper.map(SliveMapper.java:179)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.Child$4.run(Child.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.mapred.Child.main(Child.java:217)
{code}
> checkLease should throw FileNotFoundException when file does not exist
> --
>
> Key: HDFS-2484
> URL: https://issues.apache.org/jira/browse/HDFS-2484
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.22.0, 0.24.0
>Reporter: Konstantin Shvachko
>
> When file is deleted during its creation {{FSNamesystem.checkLease(String
> src, String holder)}} throws {{LeaseExpiredException}}. It would be more
> informative if it thrown {{FileNotFoundException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2484) checkLease should throw FileNotFoundException when file does not exist
checkLease should throw FileNotFoundException when file does not exist
--
Key: HDFS-2484
URL: https://issues.apache.org/jira/browse/HDFS-2484
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.22.0, 0.24.0
Reporter: Konstantin Shvachko
When file is deleted during its creation {{FSNamesystem.checkLease(String src,
String holder)}} throws {{LeaseExpiredException}}. It would be more informative
if it thrown {{FileNotFoundException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
