[jira] [Commented] (HDFS-2197) Refactor RPC call implementations out of NameNode class
[
https://issues.apache.org/jira/browse/HDFS-2197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100114#comment-13100114
]
Todd Lipcon commented on HDFS-2197:
---
This has been merged into the HA branch, but still pending merge into 0.23.
Refactor RPC call implementations out of NameNode class
---
Key: HDFS-2197
URL: https://issues.apache.org/jira/browse/HDFS-2197
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Attachments: hdfs-2197-2.txt, hdfs-2197-3.txt, hdfs-2197-4.txt,
hdfs-2197-trunk-final.txt, hdfs-2197.txt
For HA, the NameNode will gain a bit of a state machine, to be able to
transition between standby and active states. This would be cleaner in the
code if the {{NameNode}} class were just a container for various services, as
discussed in HDFS-1974. It's also nice for testing, where it would become
easier to construct just the RPC handlers around a mock NameSystem, with no
HTTP server, for example.
This JIRA is to move all of the protocol implementations out of {{NameNode}}
into a separate {{NameNodeRPCServer}} class.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100120#comment-13100120 ] Sanjay Radia commented on HDFS-2316: HDFS-2178 (HOOP) is HDFS Proxy using http protocol - a replacement for hdfs proxy v2, but providing rw access. It runs as separate daemons *typically* on an array of servers sitting next to an HDFS cluster (like HDFS proxy v2). HDFS-2316 is http rw access that replaces hftp but is *built into* the hdfs system and provides bandwidth scaling by redirecting from the NN to the datanode that contains the block. It will use spnego and delegation token. It does not requires a notion of trust. HDFS-2178 (HOOP) is run as a separate daemons that are trusted by hdfs. hdfs-2178 (HOOP) can provide additional features like bandwidth management and user authentication mapping. There is an overlap but a need for both. webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2284) Write Http access to HDFS
[ https://issues.apache.org/jira/browse/HDFS-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100124#comment-13100124 ] Sanjay Radia commented on HDFS-2284: @Sanjay Hoop as a proxy is still valuable: bandwidth management, authentication translation, etc. I has assumed that folks had understood that HDFS-2178 (HOOP) is still needed in addition to this this jira-- but I guess not. I have answered in more detail in HDFS-2316. Sorry I should have caught on earlier by Alejandro and ELi's comments: @Alejandro And yes, Hoop supports all read/write/metadata FS operations. @eli Hoop is rw already btw. Lets continue the discussion in the parent jira (2316) Write Http access to HDFS - Key: HDFS-2284 URL: https://issues.apache.org/jira/browse/HDFS-2284 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia Assignee: Tsz Wo (Nicholas), SZE Attachments: h2284_20110902b.patch, h2284_20110904_0.20s.patch, h2284_20110904b_0.20s.patch, h2284_20110905_0.20s.patch, h2284_20110906_0.20s.patch, h2284_20110906b.patch, h2284_20110906b_0.20s.patch, h2284_20110906c.patch, h2284_20110906c_0.20s.patch, h2284_20110906d.patch, h2284_20110907.patch, h2284_20110907_0.20s.patch HFTP allows on read access to HDFS via HTTP. Add write HTTP access to HDFS. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100128#comment-13100128
]
Sanjay Radia commented on HDFS-2298:
I will comment in a couple of days after reading this in more detail.
TestDfsOverAvroRpc is failing on trunk
--
Key: HDFS-2298
URL: https://issues.apache.org/jira/browse/HDFS-2298
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Aaron T. Myers
Assignee: Doug Cutting
Fix For: 0.23.0
Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
The relevant bit of the error:
{noformat}
---
Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
---
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec
FAILURE!
testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
elapsed: 1.424 sec ERROR!
org.apache.avro.AvroTypeException: Two methods with same name: delete
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2251) Namenode does not recognize incorrectly sized blocks
[ https://issues.apache.org/jira/browse/HDFS-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100254#comment-13100254 ] Uma Maheswara Rao G commented on HDFS-2251: --- Hi Brian, In which version of Hadoop you faced this? Did you observe any recovery (or) write failures? Thanks Uma Namenode does not recognize incorrectly sized blocks Key: HDFS-2251 URL: https://issues.apache.org/jira/browse/HDFS-2251 Project: Hadoop HDFS Issue Type: Bug Reporter: Brian Bockelman We had a lot of file system corruption resulting in incorrectly sized blocks (on disk, they're truncated to 192KB when they should be 64MB). However, I cannot make Hadoop realize that these blocks are incorrectly sized. When I try to drain off the node, I get the following messages: 2008-10-29 18:46:51,293 WARN org.apache.hadoop.fs.FSNamesystem: Inconsistent size for block blk_-4403534125663454855_9937 reported from 172.16.1.150:50010 current size is 67108864 reported size is 196608 Here 172.16.1.150 is not the node which has the problematic block, but the destination of the file transfer. I propose that Hadoop should either: a) Upon startup, make sure that all blocks are properly sized (pro: rather cheap check; con: doesn't catch any truncations which happen while on disk) b) Upon detecting the incorrectly sized copy, Hadoop should ask the source of the block to perform a block verification. Thanks, Brian -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2232) TestHDFSCLI fails on 0.22 branch
[ https://issues.apache.org/jira/browse/HDFS-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100275#comment-13100275 ] Hudson commented on HDFS-2232: -- Integrated in Hadoop-Hdfs-trunk #787 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/787/]) HDFS-2232. Generalize regular expressions in TestHDFSCLI. Contributed by Plamen Jeliazkov. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1166470 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml TestHDFSCLI fails on 0.22 branch Key: HDFS-2232 URL: https://issues.apache.org/jira/browse/HDFS-2232 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Plamen Jeliazkov Priority: Blocker Fix For: 0.22.0, 0.23.0, 0.24.0 Attachments: HDFS-2232-TRUNK.patch, HDFS-2232-TRUNK.patch, HDFS-2232-TRUNK.patch, HDFS-2232-TRUNK.patch, HDFS-2232.patch, HDFS-2232.patch, HDFS-2232.patch, HDFS-2232.patch, TEST-org.apache.hadoop.cli.TestHDFSCLI.txt, TEST-org.apache.hadoop.cli.TestHDFSCLI.txt, TestHDFSCLI-output.txt Several HDFS CLI tests fail on 0.22 branch. I can see 2 reasons: # Not generic enough regular expression for host names and paths. Similar to MAPREDUCE-2304. # Some command outputs have new-line in the end. # And some seem to produce [much] more output than expected. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN components
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100276#comment-13100276 ] Hudson commented on HDFS-2223: -- Integrated in Hadoop-Hdfs-trunk #787 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/787/]) HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1166466 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java Untangle depencencies between NN components --- Key: HDFS-2223 URL: https://issues.apache.org/jira/browse/HDFS-2223 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.24.0 Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2321) HDFS Build failure due to unresolved commons.daemon.os.name commons.daemon.os.arch properties in Windows 7
[
https://issues.apache.org/jira/browse/HDFS-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100280#comment-13100280
]
Uma Maheswara Rao G commented on HDFS-2321:
---
I am proceeding with below conditions in my local environment as a workaround.
condition property=commons.daemon.os.name value=linux
os family=windows/
/condition
condition property=commons.daemon.os.arch value=i686
os family=windows/
/condition
Here our building env is in windows, because of that we are getting this error
for every updation.
Alejandro, Do you have any better suggestion here to handle this?
it will be problem if target os is solaris and building env is windows.
Thanks
Uma
HDFS Build failure due to unresolved commons.daemon.os.name
commons.daemon.os.arch properties in Windows 7
Key: HDFS-2321
URL: https://issues.apache.org/jira/browse/HDFS-2321
Project: Hadoop HDFS
Issue Type: Bug
Components: build
Affects Versions: 0.24.0
Environment: Windows 7, Windows XP
Reporter: Uma Maheswara Rao G
This is the script for resolving the commons-deamons***tar.gz in hdfs pom.xml.
target
condition property=commons.daemon.os.name value=darwin
os name=Mac OS X/
/condition
condition property=commons.daemon.os.arch
value=universal
os name=Mac OS X/
/condition
condition property=commons.daemon.os.name value=linux
os name=Linux /
/condition
!-- Set commons.daemon.os.arch to either i686 or x86_64 for
GNU/Linux --
condition property=commons.daemon.os.arch value=x86_64
os name=Linux arch=amd64/
/condition
condition property=commons.daemon.os.arch value=i686
os name=Linux / !-- This is a guess --
/condition
property name=commons.daemon.tar.name
value=commons-daemon-${commons-daemon.version}-bin-${commons.daemon.os.name}-${commons.daemon.os.arch}.tar.gz/
Here we are not handling for windows.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2223) Untangle depencencies between NN components
[ https://issues.apache.org/jira/browse/HDFS-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100295#comment-13100295 ] Hudson commented on HDFS-2223: -- Integrated in Hadoop-Mapreduce-trunk #810 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/810/]) HDFS-2223. Untangle depencencies between NN components. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1166466 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeResourceChecker.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/common/TestDistributedUpgrade.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java Untangle depencencies between NN components --- Key: HDFS-2223 URL: https://issues.apache.org/jira/browse/HDFS-2223 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.24.0 Attachments: hdfs-2223-1.txt, hdfs-2223-10.txt, hdfs-2223-2.txt, hdfs-2223-3.txt, hdfs-2223-4.txt, hdfs-2223-5.txt, hdfs-2223-6.txt, hdfs-2223-7.txt, hdfs-2223-8.txt, hdfs-2223-9.txt Working in the NN a lot for HA (HDFS-1623) I've come across a number of situations where the tangled dependencies between NN components has been problematic for adding new features and for testability. It would be good to untangle some of these and clarify what the distinction is between the different components: NameNode, FSNamesystem, FSDirectory, FSImage, NNStorage, and FSEditLog -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2232) TestHDFSCLI fails on 0.22 branch
[ https://issues.apache.org/jira/browse/HDFS-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100294#comment-13100294 ] Hudson commented on HDFS-2232: -- Integrated in Hadoop-Mapreduce-trunk #810 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/810/]) HDFS-2232. Generalize regular expressions in TestHDFSCLI. Contributed by Plamen Jeliazkov. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1166470 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml TestHDFSCLI fails on 0.22 branch Key: HDFS-2232 URL: https://issues.apache.org/jira/browse/HDFS-2232 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Plamen Jeliazkov Priority: Blocker Fix For: 0.22.0, 0.23.0, 0.24.0 Attachments: HDFS-2232-TRUNK.patch, HDFS-2232-TRUNK.patch, HDFS-2232-TRUNK.patch, HDFS-2232-TRUNK.patch, HDFS-2232.patch, HDFS-2232.patch, HDFS-2232.patch, HDFS-2232.patch, TEST-org.apache.hadoop.cli.TestHDFSCLI.txt, TEST-org.apache.hadoop.cli.TestHDFSCLI.txt, TestHDFSCLI-output.txt Several HDFS CLI tests fail on 0.22 branch. I can see 2 reasons: # Not generic enough regular expression for host names and paths. Similar to MAPREDUCE-2304. # Some command outputs have new-line in the end. # And some seem to produce [much] more output than expected. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2321) HDFS Build failure due to unresolved commons.daemon.os.name commons.daemon.os.arch properties in Windows 7
[
https://issues.apache.org/jira/browse/HDFS-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100299#comment-13100299
]
Uma Maheswara Rao G commented on HDFS-2321:
---
-D options also would work fine.
E:\Trunk-Commonmvn clean install -DskipTests -Dcommons.daemon.os.arch=i686 -Dco
mmons.daemon.os.name=linux
HDFS Build failure due to unresolved commons.daemon.os.name
commons.daemon.os.arch properties in Windows 7
Key: HDFS-2321
URL: https://issues.apache.org/jira/browse/HDFS-2321
Project: Hadoop HDFS
Issue Type: Bug
Components: build
Affects Versions: 0.24.0
Environment: Windows 7, Windows XP
Reporter: Uma Maheswara Rao G
This is the script for resolving the commons-deamons***tar.gz in hdfs pom.xml.
target
condition property=commons.daemon.os.name value=darwin
os name=Mac OS X/
/condition
condition property=commons.daemon.os.arch
value=universal
os name=Mac OS X/
/condition
condition property=commons.daemon.os.name value=linux
os name=Linux /
/condition
!-- Set commons.daemon.os.arch to either i686 or x86_64 for
GNU/Linux --
condition property=commons.daemon.os.arch value=x86_64
os name=Linux arch=amd64/
/condition
condition property=commons.daemon.os.arch value=i686
os name=Linux / !-- This is a guess --
/condition
property name=commons.daemon.tar.name
value=commons-daemon-${commons-daemon.version}-bin-${commons.daemon.os.name}-${commons.daemon.os.arch}.tar.gz/
Here we are not handling for windows.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-197) du fails on Cygwin
[
https://issues.apache.org/jira/browse/HDFS-197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J reassigned HDFS-197:
Assignee: Harsh J (was: Kohsuke Kawaguchi)
du fails on Cygwin
Key: HDFS-197
URL: https://issues.apache.org/jira/browse/HDFS-197
Project: Hadoop HDFS
Issue Type: Bug
Environment: Windows + Cygwin
Reporter: Kohsuke Kawaguchi
Assignee: Harsh J
Attachments: HADOOP-5486
When I try to run a datanode on Windows, I get the following exception:
{noformat}
java.io.IOException: Expecting a line not the end of stream
at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DU.init(DU.java:53)
at org.apache.hadoop.fs.DU.init(DU.java:63)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.init(FSDataset.java:325)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.init(FSDataset.java:681)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:205)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
{noformat}
This is because Hadoop execs du -sk C:\tmp\hadoop-SYSTEM\dfs\data with a
Windows path representation, which cygwin du doesn't understand.
{noformat}
C:\hudsondu -sk C:\tmp\hadoop-SYSTEM\dfs\data
du -sk C:\tmp\hadoop-SYSTEM\dfs\data
du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or
directory
{noformat}
For this to work correctly, Hadoop would have to run cygpath first to get a
Unix path representation, then to call DU.
Also, I had to use the debugger to get this information. Shell.runCommand
should catch IOException from parseExecResult and add the buffered stderr to
simplify the error diagnostics.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-197) du fails on Cygwin
[
https://issues.apache.org/jira/browse/HDFS-197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100325#comment-13100325
]
Harsh J commented on HDFS-197:
--
(Assigning to self, will try a test case and update patch accordingly.)
du fails on Cygwin
Key: HDFS-197
URL: https://issues.apache.org/jira/browse/HDFS-197
Project: Hadoop HDFS
Issue Type: Bug
Environment: Windows + Cygwin
Reporter: Kohsuke Kawaguchi
Assignee: Harsh J
Attachments: HADOOP-5486
When I try to run a datanode on Windows, I get the following exception:
{noformat}
java.io.IOException: Expecting a line not the end of stream
at org.apache.hadoop.fs.DU.parseExecResult(DU.java:181)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:179)
at org.apache.hadoop.util.Shell.run(Shell.java:134)
at org.apache.hadoop.fs.DU.init(DU.java:53)
at org.apache.hadoop.fs.DU.init(DU.java:63)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.init(FSDataset.java:325)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.init(FSDataset.java:681)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:291)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:205)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1238)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1193)
{noformat}
This is because Hadoop execs du -sk C:\tmp\hadoop-SYSTEM\dfs\data with a
Windows path representation, which cygwin du doesn't understand.
{noformat}
C:\hudsondu -sk C:\tmp\hadoop-SYSTEM\dfs\data
du -sk C:\tmp\hadoop-SYSTEM\dfs\data
du: cannot access `C:\\tmp\\hadoop-SYSTEM\\dfs\\data': No such file or
directory
{noformat}
For this to work correctly, Hadoop would have to run cygpath first to get a
Unix path representation, then to call DU.
Also, I had to use the debugger to get this information. Shell.runCommand
should catch IOException from parseExecResult and add the buffered stderr to
simplify the error diagnostics.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2271: -- Component/s: name-node startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception. --- Key: HDFS-2271 URL: https://issues.apache.org/jira/browse/HDFS-2271 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Attachments: HDFS-2271.patch Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2060) DFS client RPCs using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100466#comment-13100466 ] Doug Cutting commented on HDFS-2060: I agree that the MR2-style protocol wrappers are a good pattern, but I'm not convinced they're required before this could be committed. The current Writable-based RPC implementation is not wrapped, nor does it provide version interoperability. This patch addresses version-interoperability but not wrapping. Wrapping seems like it might be handled as an independent issue, especially if version-interoperability is a priority in the near term. DFS client RPCs using protobufs --- Key: HDFS-2060 URL: https://issues.apache.org/jira/browse/HDFS-2060 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-2060-getblocklocations.txt The most important place for wire-compatibility in DFS is between clients and the cluster, since lockstep upgrade is very difficult and a single client may want to talk to multiple server versions. So, I'd like to focus this JIRA on making the RPCs between the DFS client and the NN/DNs wire-compatible using protocol buffer based serialization. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1242) 0.20 append: Add test for appendFile() race solved in HDFS-142
[ https://issues.apache.org/jira/browse/HDFS-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1242: --- Fix Version/s: 0.20.205.0 0.20 append: Add test for appendFile() race solved in HDFS-142 -- Key: HDFS-1242 URL: https://issues.apache.org/jira/browse/HDFS-1242 Project: Hadoop HDFS Issue Type: Test Affects Versions: 0.20-append Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.20-append, 0.20.205.0 Attachments: hdfs-1242.txt This is a unit test that didn't make it into branch-0.20-append, but worth having in TestFileAppend4. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1242) 0.20 append: Add test for appendFile() race solved in HDFS-142
[ https://issues.apache.org/jira/browse/HDFS-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100577#comment-13100577 ] Jitendra Nath Pandey commented on HDFS-1242: +1. It is a useful test. 0.20 append: Add test for appendFile() race solved in HDFS-142 -- Key: HDFS-1242 URL: https://issues.apache.org/jira/browse/HDFS-1242 Project: Hadoop HDFS Issue Type: Test Affects Versions: 0.20-append Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.20-append, 0.20.205.0 Attachments: hdfs-1242.txt This is a unit test that didn't make it into branch-0.20-append, but worth having in TestFileAppend4. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2322) the build fails in Windows because commons-daemon TAR cannot be fetched
the build fails in Windows because commons-daemon TAR cannot be fetched --- Key: HDFS-2322 URL: https://issues.apache.org/jira/browse/HDFS-2322 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 For windows there is no commons-daemon TAR but a ZIP, plus the name follows a different convention. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2322) the build fails in Windows because commons-daemon TAR cannot be fetched
[ https://issues.apache.org/jira/browse/HDFS-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100579#comment-13100579 ] Alejandro Abdelnur commented on HDFS-2322: -- As native stuff is not supported on windows, we should just skip the downloading of the commons-daemons TAR the build fails in Windows because commons-daemon TAR cannot be fetched --- Key: HDFS-2322 URL: https://issues.apache.org/jira/browse/HDFS-2322 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 For windows there is no commons-daemon TAR but a ZIP, plus the name follows a different convention. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2322) the build fails in Windows because commons-daemon TAR cannot be fetched
[ https://issues.apache.org/jira/browse/HDFS-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-2322: - Status: Patch Available (was: Open) the build fails in Windows because commons-daemon TAR cannot be fetched --- Key: HDFS-2322 URL: https://issues.apache.org/jira/browse/HDFS-2322 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: HDFS-2322v1.patch For windows there is no commons-daemon TAR but a ZIP, plus the name follows a different convention. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2322) the build fails in Windows because commons-daemon TAR cannot be fetched
[ https://issues.apache.org/jira/browse/HDFS-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-2322: - Attachment: HDFS-2322v1.patch the build fails in Windows because commons-daemon TAR cannot be fetched --- Key: HDFS-2322 URL: https://issues.apache.org/jira/browse/HDFS-2322 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: HDFS-2322v1.patch For windows there is no commons-daemon TAR but a ZIP, plus the name follows a different convention. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2322) the build fails in Windows because commons-daemon TAR cannot be fetched
[ https://issues.apache.org/jira/browse/HDFS-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100594#comment-13100594 ] Uma Maheswara Rao G commented on HDFS-2322: --- Duplicate of HDFS-2321 the build fails in Windows because commons-daemon TAR cannot be fetched --- Key: HDFS-2322 URL: https://issues.apache.org/jira/browse/HDFS-2322 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: HDFS-2322v1.patch For windows there is no commons-daemon TAR but a ZIP, plus the name follows a different convention. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1242) 0.20 append: Add test for appendFile() race solved in HDFS-142
[ https://issues.apache.org/jira/browse/HDFS-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100596#comment-13100596 ] Jitendra Nath Pandey commented on HDFS-1242: I have committed to 20-security. Thanks to Todd. 0.20 append: Add test for appendFile() race solved in HDFS-142 -- Key: HDFS-1242 URL: https://issues.apache.org/jira/browse/HDFS-1242 Project: Hadoop HDFS Issue Type: Test Affects Versions: 0.20-append Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.20-append, 0.20.205.0 Attachments: hdfs-1242.txt This is a unit test that didn't make it into branch-0.20-append, but worth having in TestFileAppend4. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2321) HDFS Build failure due to unresolved commons.daemon.os.name commons.daemon.os.arch properties in Windows 7
[
https://issues.apache.org/jira/browse/HDFS-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100625#comment-13100625
]
Alejandro Abdelnur commented on HDFS-2321:
--
Uma, I wouldn't force the os.arch/os.name to linux, instead I'd skip that
download as per patch HDFS-2322
HDFS Build failure due to unresolved commons.daemon.os.name
commons.daemon.os.arch properties in Windows 7
Key: HDFS-2321
URL: https://issues.apache.org/jira/browse/HDFS-2321
Project: Hadoop HDFS
Issue Type: Bug
Components: build
Affects Versions: 0.24.0
Environment: Windows 7, Windows XP
Reporter: Uma Maheswara Rao G
This is the script for resolving the commons-deamons***tar.gz in hdfs pom.xml.
target
condition property=commons.daemon.os.name value=darwin
os name=Mac OS X/
/condition
condition property=commons.daemon.os.arch
value=universal
os name=Mac OS X/
/condition
condition property=commons.daemon.os.name value=linux
os name=Linux /
/condition
!-- Set commons.daemon.os.arch to either i686 or x86_64 for
GNU/Linux --
condition property=commons.daemon.os.arch value=x86_64
os name=Linux arch=amd64/
/condition
condition property=commons.daemon.os.arch value=i686
os name=Linux / !-- This is a guess --
/condition
property name=commons.daemon.tar.name
value=commons-daemon-${commons-daemon.version}-bin-${commons.daemon.os.name}-${commons.daemon.os.arch}.tar.gz/
Here we are not handling for windows.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2322) the build fails in Windows because commons-daemon TAR cannot be fetched
[ https://issues.apache.org/jira/browse/HDFS-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100647#comment-13100647 ] Hadoop QA commented on HDFS-2322: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493640/HDFS-2322v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDfsOverAvroRpc org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap org.apache.hadoop.hdfs.server.datanode.TestReplicasMap +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1224//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1224//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1224//console This message is automatically generated. the build fails in Windows because commons-daemon TAR cannot be fetched --- Key: HDFS-2322 URL: https://issues.apache.org/jira/browse/HDFS-2322 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 0.23.0, 0.24.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0, 0.24.0 Attachments: HDFS-2322v1.patch For windows there is no commons-daemon TAR but a ZIP, plus the name follows a different convention. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1122) client block verification may result in blocks in DataBlockScanner prematurely
[ https://issues.apache.org/jira/browse/HDFS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1122: --- Attachment: HDFS-1122-20-security.2.patch The attached patch is Sam's original patch with following changes 1) Rebased against 20-security. 2) Changed testcase to junit4. 3) Addressed Todd's comments. client block verification may result in blocks in DataBlockScanner prematurely -- Key: HDFS-1122 URL: https://issues.apache.org/jira/browse/HDFS-1122 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20-append Reporter: sam rash Assignee: sam rash Attachments: HDFS-1122-20-security.2.patch, hdfs-1122-for-0.20.txt found that when the DN uses client verification of a block that is open for writing, it will add it to the DataBlockScanner prematurely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1122) client block verification may result in blocks in DataBlockScanner prematurely
[ https://issues.apache.org/jira/browse/HDFS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1122: --- Affects Version/s: 0.20.205.0 Fix Version/s: 0.20.205.0 0.20-append client block verification may result in blocks in DataBlockScanner prematurely -- Key: HDFS-1122 URL: https://issues.apache.org/jira/browse/HDFS-1122 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20-append, 0.20.205.0 Reporter: sam rash Assignee: sam rash Fix For: 0.20-append, 0.20.205.0 Attachments: HDFS-1122-20-security.2.patch, hdfs-1122-for-0.20.txt found that when the DN uses client verification of a block that is open for writing, it will add it to the DataBlockScanner prematurely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1122) client block verification may result in blocks in DataBlockScanner prematurely
[ https://issues.apache.org/jira/browse/HDFS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100671#comment-13100671 ] Suresh Srinivas commented on HDFS-1122: --- +1 for the patch. client block verification may result in blocks in DataBlockScanner prematurely -- Key: HDFS-1122 URL: https://issues.apache.org/jira/browse/HDFS-1122 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20-append, 0.20.205.0 Reporter: sam rash Assignee: sam rash Fix For: 0.20-append, 0.20.205.0 Attachments: HDFS-1122-20-security.2.patch, hdfs-1122-for-0.20.txt found that when the DN uses client verification of a block that is open for writing, it will add it to the DataBlockScanner prematurely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly
[ https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100675#comment-13100675 ] Sanjay Radia commented on HDFS-2246: getBlockPathInfo(block) - add block access token parameter which is verified on the data node. Shortcut a local client reads to a Datanodes files directly --- Key: HDFS-2246 URL: https://issues.apache.org/jira/browse/HDFS-2246 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Attachments: 0001-HDFS-347.-Local-reads.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2320) Make merged protocol changes from 0.20-append to 0.20-security compatible with previous releases.
[ https://issues.apache.org/jira/browse/HDFS-2320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2320: -- Attachment: HDFS-2320.patch Updated patch to remove an unnecessary file change. I also tested this, killing datanodes while running TestDFSIO Make merged protocol changes from 0.20-append to 0.20-security compatible with previous releases. - Key: HDFS-2320 URL: https://issues.apache.org/jira/browse/HDFS-2320 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client, name-node Affects Versions: 0.20.205.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.20.205.0 Attachments: HDFS-2320.patch, HDFS-2320.patch 0.20-append changes have been merged to 0.20-security. The merge has changes to version numbers in several protocols. This jira makes the protocol changes compatible with older release, allowing clients running older version to talk to server running 205 version and clients running 205 version talk to older servers running 203, 204. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2323) start-dfs.sh script fails for tarball install
start-dfs.sh script fails for tarball install
-
Key: HDFS-2323
URL: https://issues.apache.org/jira/browse/HDFS-2323
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Tom White
Assignee: Tom White
I build Common and HDFS tarballs from trunk then tried to start a cluster with
start-dfs.sh, but I got the following error:
{noformat}
Starting namenodes on [localhost ]
sbin/start-dfs.sh: line 55:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
sbin/start-dfs.sh: line 68:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
Starting secondary namenodes [0.0.0.0 ]
sbin/start-dfs.sh: line 88:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2323) start-dfs.sh script fails for tarball install
[
https://issues.apache.org/jira/browse/HDFS-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HDFS-2323:
Attachment: HDFS-2323.patch
There are a couple of problems:
1. The scripts refer to bin/hadoop-daemons.sh not sbin/hadoop-daemons.sh.
2. The secondary namenode should not be started at all since it's not
configured to, but the script tries to start it since the test for 0.0.0.0
fails due to a trailing space.
This patch fixes both of these.
start-dfs.sh script fails for tarball install
-
Key: HDFS-2323
URL: https://issues.apache.org/jira/browse/HDFS-2323
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Tom White
Assignee: Tom White
Attachments: HDFS-2323.patch
I build Common and HDFS tarballs from trunk then tried to start a cluster
with start-dfs.sh, but I got the following error:
{noformat}
Starting namenodes on [localhost ]
sbin/start-dfs.sh: line 55:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
sbin/start-dfs.sh: line 68:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
Starting secondary namenodes [0.0.0.0 ]
sbin/start-dfs.sh: line 88:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2323) start-dfs.sh script fails for tarball install
[
https://issues.apache.org/jira/browse/HDFS-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100782#comment-13100782
]
Tom White commented on HDFS-2323:
-
To ran a pseudo distributed HDFS cluster I ran
{noformat}
mvn clean package -Pdist -Dtar -DskipTests -P-cbuild
{noformat}
from the top level, then from another directory
{noformat}
mkdir hadoop
cd hadoop
tar zxf
~/workspace/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-0.24.0-SNAPSHOT.tar.gz
--strip 1
tar zxf
~/workspace/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-0.24.0-SNAPSHOT.tar.gz
--strip 1
export HADOOP_COMMON_HOME=$(pwd)
export HADOOP_HDFS_HOME=$(pwd)
cat $HADOOP_COMMON_HOME/etc/hadoop/core-site.xml EOF
?xml version=1.0?
!-- core-site.xml --
configuration
property
namefs.default.name/name
valuehdfs://localhost//value
/property
/configuration
EOF
sbin/start-dfs.sh
{noformat}
start-dfs.sh script fails for tarball install
-
Key: HDFS-2323
URL: https://issues.apache.org/jira/browse/HDFS-2323
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Tom White
Assignee: Tom White
Attachments: HDFS-2323.patch
I build Common and HDFS tarballs from trunk then tried to start a cluster
with start-dfs.sh, but I got the following error:
{noformat}
Starting namenodes on [localhost ]
sbin/start-dfs.sh: line 55:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
sbin/start-dfs.sh: line 68:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
Starting secondary namenodes [0.0.0.0 ]
sbin/start-dfs.sh: line 88:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2323) start-dfs.sh script fails for tarball install
[
https://issues.apache.org/jira/browse/HDFS-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated HDFS-2323:
Status: Patch Available (was: Open)
start-dfs.sh script fails for tarball install
-
Key: HDFS-2323
URL: https://issues.apache.org/jira/browse/HDFS-2323
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Tom White
Assignee: Tom White
Attachments: HDFS-2323.patch
I build Common and HDFS tarballs from trunk then tried to start a cluster
with start-dfs.sh, but I got the following error:
{noformat}
Starting namenodes on [localhost ]
sbin/start-dfs.sh: line 55:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
sbin/start-dfs.sh: line 68:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
Starting secondary namenodes [0.0.0.0 ]
sbin/start-dfs.sh: line 88:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100797#comment-13100797 ] Eli Collins commented on HDFS-2316: --- I definitely agree with these distinct use cases, I'm just wondering if we need to have two separate FileSystem over HTTP implementations vs one client that may or may not use a proxy server (there's no reason http or FileSystem clients need to care whether they're being proxied). Sounds like we were duplicating code w/o understanding what could be shared. You and Alejandro have looked at the specifics more than I have so I trust your judgement. webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2320) Make merged protocol changes from 0.20-append to 0.20-security compatible with previous releases.
[ https://issues.apache.org/jira/browse/HDFS-2320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100801#comment-13100801 ] Jitendra Nath Pandey commented on HDFS-2320: +1. The patch looks good to me. Make merged protocol changes from 0.20-append to 0.20-security compatible with previous releases. - Key: HDFS-2320 URL: https://issues.apache.org/jira/browse/HDFS-2320 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client, name-node Affects Versions: 0.20.205.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.20.205.0 Attachments: HDFS-2320.patch, HDFS-2320.patch 0.20-append changes have been merged to 0.20-security. The merge has changes to version numbers in several protocols. This jira makes the protocol changes compatible with older release, allowing clients running older version to talk to server running 205 version and clients running 205 version talk to older servers running 203, 204. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1122) client block verification may result in blocks in DataBlockScanner prematurely
[ https://issues.apache.org/jira/browse/HDFS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100804#comment-13100804 ] Jitendra Nath Pandey commented on HDFS-1122: Committed to branch-0.20-security. Thanks to Sam. client block verification may result in blocks in DataBlockScanner prematurely -- Key: HDFS-1122 URL: https://issues.apache.org/jira/browse/HDFS-1122 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20-append, 0.20.205.0 Reporter: sam rash Assignee: sam rash Fix For: 0.20-append, 0.20.205.0 Attachments: HDFS-1122-20-security.2.patch, hdfs-1122-for-0.20.txt found that when the DN uses client verification of a block that is open for writing, it will add it to the DataBlockScanner prematurely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2320) Make merged protocol changes from 0.20-append to 0.20-security compatible with previous releases.
[ https://issues.apache.org/jira/browse/HDFS-2320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-2320. --- Resolution: Fixed Hadoop Flags: [Reviewed] I committed this patch. This patch is only relevant to 0.20.205, due to merging 0.20-append changes and is not relevant to trunk which has a new implementation of append. Make merged protocol changes from 0.20-append to 0.20-security compatible with previous releases. - Key: HDFS-2320 URL: https://issues.apache.org/jira/browse/HDFS-2320 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client, name-node Affects Versions: 0.20.205.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.20.205.0 Attachments: HDFS-2320.patch, HDFS-2320.patch 0.20-append changes have been merged to 0.20-security. The merge has changes to version numbers in several protocols. This jira makes the protocol changes compatible with older release, allowing clients running older version to talk to server running 205 version and clients running 205 version talk to older servers running 203, 204. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100816#comment-13100816 ] Sanjay Radia commented on HDFS-2316: One issue: we have to co exist with hftp which uses a prefixes like /data. Two proposal on the table: # (by Nicholas) use a different fixed prefix like /webhdfs/path since it does not conflict with /data. # (by Alejandro) always use an operation in the url so that one can send /data with an operation to webhdfs and /data without operation to hftp. 1) is is simpler to implement since urls within the context of /webhdfs can be sent to the webhdfs servlet. 2)has a nicer url since the path is the pathname being refereced. webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2323) start-dfs.sh script fails for tarball install
[
https://issues.apache.org/jira/browse/HDFS-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100836#comment-13100836
]
Hadoop QA commented on HDFS-2323:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493689/HDFS-2323.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.TestDfsOverAvroRpc
org.apache.hadoop.hdfs.server.blockmanagement.TestHost2NodesMap
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/1225//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/1225//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1225//console
This message is automatically generated.
start-dfs.sh script fails for tarball install
-
Key: HDFS-2323
URL: https://issues.apache.org/jira/browse/HDFS-2323
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Tom White
Assignee: Tom White
Attachments: HDFS-2323.patch
I build Common and HDFS tarballs from trunk then tried to start a cluster
with start-dfs.sh, but I got the following error:
{noformat}
Starting namenodes on [localhost ]
sbin/start-dfs.sh: line 55:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
sbin/start-dfs.sh: line 68:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
Starting secondary namenodes [0.0.0.0 ]
sbin/start-dfs.sh: line 88:
/Users/tom/tmp/hadoop/libexec/../bin/hadoop-daemons.sh: No such file or
directory
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100862#comment-13100862 ] Tsz Wo (Nicholas), SZE commented on HDFS-2316: -- 2) may be confusion since the same http url path represents two different file system paths. E.g. - http://namenode:port/data/foo/bar?... means reading /foo/bar - http://namenode:port/data/foo/bar?opGet=read;... means reading /data/foo/bar webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100891#comment-13100891 ] Alejandro Abdelnur commented on HDFS-2316: -- IMO, the nice thing about #2 is that the file path of *HDFS:* and a *HTTP:* URIs will be exactly the same, and in the case of using the NN/DD deployment of HOOP it will be even the same host. In addition is it intuitive without any caveat, a given path will just work by replacing the SCHEME://HOST:PORT part of it. Finally, and IMO this is very important from the Usability perspective, user applications that take are designed to take the URI of the FS as parameter and operate via HDFS: or HTTP: will be otherwise difficult to code. Hadoop's *Path(String parent, String child)* uses the *URI.resolve(...)* that uses a well defined logic to resolve URIs based on other URIs[ http://download.oracle.com/javase/6/docs/api/java/net/URI.html#resolve(java.net.URI) ]. If we use a prefix for HTTP URIs then it will become difficult and error prone to compose HDFS: URIs from HTTP: URIs and viceversa. (And I believe the same is true for libraries in other languages) Finally, I have not seen HDFS files under */data* as a common practice, thus the name collision won't be that common. webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100917#comment-13100917 ] Todd Lipcon commented on HDFS-2316: --- Maybe I'm not following completely: is the idea that webhdfs would run on the same port as the NN web UI? It seems crazy to me that http://nn:50030/jmx would be the JMX servlet but http://nn:50030/jmx?opGet=read would be a file at path /jmx... hopefully I'm misunderstanding :) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100926#comment-13100926 ] Tsz Wo (Nicholas), SZE commented on HDFS-2316: -- @Todd, we are using the same port. If we implement (2), you are right that we will have such problem. webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100934#comment-13100934 ] Todd Lipcon commented on HDFS-2316: --- Yea, that seems crazy. I'm for option #1 or for opening yet-another-port. webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100960#comment-13100960 ] Sanjay Radia commented on HDFS-2316: Operations like to minimize the number of ports. So we would use the same port. webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100965#comment-13100965 ] Sanjay Radia commented on HDFS-2316: Like hftp, Operations are processed at the NN if they involve no data transfer. If the operation involves data transfer (r or w) then the request is redirected to the DN. This allows bandwidth scaling and load distribution. Alejandro has pointed out that when you redirect a put or post operation, the initial part of the payload has been sent to the NN. I believe this is true. Hence for writes we could consider a two request mode - getTheWrite handle using a get and then do a put or post to the data node. webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100978#comment-13100978 ] Alejandro Abdelnur commented on HDFS-2316: -- @Todd, well the problem here is that we are overloading the use of a PORT for a functionality that requires the whole domain of the namespace (in HFDS we don't have special dirs like /dev). I'd be OK with a different port then. @Sanjay, the in-direction for writes it would mean that we moving away from a the REST protocol, which is a well understood and known way of interacting with resources. I think there a big value in having a full REST API like Hoop has today. webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13100993#comment-13100993 ] Alejandro Abdelnur commented on HDFS-2316: -- If we use a webhdfs://HOST:PORT from the FS client impl and internally we replace to it to http://HOST:PORT/webhdfs, I'd live with that one. After all (as Todd pointed offline) it is not fully REST. But the write/append handle thingy, any other option? webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP -- Key: HDFS-2316 URL: https://issues.apache.org/jira/browse/HDFS-2316 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE We current have hftp for accessing HDFS over HTTP. However, hftp is a read-only FileSystem and does not provide write accesses. In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem implementation for accessing HDFS over HTTP. The is the umbrella JIRA for the tasks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[
https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101001#comment-13101001
]
Tsz Wo (Nicholas), SZE commented on HDFS-2178:
--
Nicholas, thank you very much for taking the time to go through Hoop's code
and thanks for your positive feedback on it.
Regarding create mkdir being POST operation.
...
Hi Alejandro, I still think that we should use PUT for mkdir and create. Below
is quoted from Section 9.6 PUT in
http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html
{quote}
The fundamental difference between the POST and PUT requests is reflected in
the different meaning of the Request-URI. The URI in a POST request identifies
the resource that will handle the enclosed entity. That resource might be a
data-accepting process, a gateway to some other protocol, or a separate entity
that accepts annotations. In contrast, the URI in a PUT request identifies the
entity enclosed with the request -- the user agent knows what URI is intended
and the server MUST NOT attempt to apply the request to some other resource. ...
{quote}
Contributing Hoop to HDFS, replacement for HDFS proxy with read/write
capabilities
--
Key: HDFS-2178
URL: https://issues.apache.org/jira/browse/HDFS-2178
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 0.23.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 0.23.0
We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement)
for HDFS Proxy.
Hoop provides access to all Hadoop Distributed File System (HDFS) operations
(read and write) over HTTP/S.
The Hoop server component is a REST HTTP gateway to HDFS supporting all file
system operations. It can be accessed using standard HTTP tools (i.e. curl
and wget), HTTP libraries from different programing languages (i.e. Perl,
Java Script) as well as using the Hoop client. The Hoop server component is a
standard Java web-application and it has been implemented using Jersey
(JAX-RS).
The Hoop client component is an implementation of Hadoop FileSystem client
that allows using the familiar Hadoop filesystem API to access HDFS data
through a Hoop server.
Repo: https://github.com/cloudera/hoop
Docs: http://cloudera.github.com/hoop
Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/
Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for
Kerberos HTTP SPNEGO authentication).
To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be
done first, as well as the Alfredo contribution (HADOOP-7119).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1779) After NameNode restart , Clients can not read partial files even after client invokes Sync.
[ https://issues.apache.org/jira/browse/HDFS-1779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13101005#comment-13101005 ] Jitendra Nath Pandey commented on HDFS-1779: +1 for the patch. This needs to be ported to 20-security branch as well. After NameNode restart , Clients can not read partial files even after client invokes Sync. --- Key: HDFS-1779 URL: https://issues.apache.org/jira/browse/HDFS-1779 Project: Hadoop HDFS Issue Type: Bug Components: data-node, name-node Affects Versions: 0.20-append Environment: Linux Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Fix For: 0.20-append, 0.20.205.0 Attachments: HDFS-1779.1.patch, HDFS-1779.2.patch, HDFS-1779.2a.patch, HDFS-1779.patch, bbwReportAppend.patch In Append HDFS-200 issue, If file has 10 blocks and after writing 5 blocks if client invokes sync method then NN will persist the blocks information in edits. After this if we restart the NN, All the DataNodes will reregister with NN. But DataNodes are not sending the blocks being written information to NN. DNs are sending the blocksBeingWritten information in DN startup. So, here NameNode can not find that the 5 persisted blocks belongs to which datanodes. This information can build based on block reports from DN. Otherwise we will loose this 5 blocks information even NN persisted that block information in edits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
