[jira] [Updated] (HDFS-1619) Remove AC_TYPE* from the libhdfs
[ https://issues.apache.org/jira/browse/HDFS-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-1619: -- Resolution: Fixed Status: Resolved (was: Patch Available) I've committed this to branch 22 and trunk. Thanks Roman! Remove AC_TYPE* from the libhdfs Key: HDFS-1619 URL: https://issues.apache.org/jira/browse/HDFS-1619 Project: Hadoop HDFS Issue Type: Improvement Components: libhdfs Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Fix For: 0.22.0 Attachments: HDFS-1619-C99.patch.txt, HDFS-1619.patch.txt, hdfs-1619-2.patch Remove AC_TYPE* from the libhdfs build since we get these via stdint. Currently configure.ac uses AC_TYPE_INT16_T, AC_TYPE_INT32_T, AC_TYPE_INT64_T and AC_TYPE_UINT16_T and thus requires autoconf 2.61 or higher. This prevents using it on such platforms as CentOS/RHEL 5.4 and 5.5. Given that those are pretty popular and also given that it is really difficult to find a platform these days that doesn't natively define intXX_t types I'm curious as to whether we can simply remove those macros or perhaps fail ONLY if we happen to be on such a platform. Here's a link to GNU autoconf docs for your reference: http://www.gnu.org/software/hello/manual/autoconf/Particular-Types.html -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2004) Enable replicating and pinning files to a data node
[ https://issues.apache.org/jira/browse/HDFS-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved HDFS-2004. -- Resolution: Won't Fix bq. You are certainly entitled to your opinion. I'm also entitled to mine. -1 it is. Duly noted. Closing as won't fix because of Allen's veto. @Jason: You should obviously feel free to develop this feature if you'd like. If you'd like to post a patch here you're welcome to. However, you should do so with the understanding that it will not be committed to any branch of HDFS. Enable replicating and pinning files to a data node --- Key: HDFS-2004 URL: https://issues.apache.org/jira/browse/HDFS-2004 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Affects Versions: 0.23.0 Reporter: Jason Rutherglen Some HDFS applications require that a given file is on the local DataNode. The functionality created here will allow pinning the file to any DataNode. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2028) There are lot of ESTABLISHED socket connetions at 50010 port.
[ https://issues.apache.org/jira/browse/HDFS-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045338#comment-13045338 ] Zhou Sheng commented on HDFS-2028: -- Thanks for your reply, and sorry for my late reply. Because the past three days are the Chinese Dragon Boat Festival. About HDFS-1836 I have tested the v0.20.3 and the problem continues. Because I saw the fixed version is 0.20.3 and 0.20.205.0. OK, I will test for the version 0.20.205.0. Thank you again! There are lot of ESTABLISHED socket connetions at 50010 port. - Key: HDFS-2028 URL: https://issues.apache.org/jira/browse/HDFS-2028 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.20.3 Environment: linux server Reporter: Zhou Sheng Labels: hadoop When I load data into tables of hive via hiver cli or hive server, there are always some ESTABLISHED or CLOSE_WAIT status socket connections. And these tcp connections won't be released unless you quit hive or restart hiveserver. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1619) Remove AC_TYPE* from the libhdfs
[ https://issues.apache.org/jira/browse/HDFS-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045376#comment-13045376 ] Hudson commented on HDFS-1619: -- Integrated in Hadoop-Hdfs-22-branch #63 (See [https://builds.apache.org/job/Hadoop-Hdfs-22-branch/63/]) HDFS-1619. svn merge -c 1132881 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1132883 Files : * /hadoop/hdfs/branches/branch-0.22/src/test/hdfs * /hadoop/hdfs/branches/branch-0.22 * /hadoop/hdfs/branches/branch-0.22/src/webapps/secondary * /hadoop/hdfs/branches/branch-0.22/src/java * /hadoop/hdfs/branches/branch-0.22/src/webapps/hdfs * /hadoop/hdfs/branches/branch-0.22/src/webapps/datanode * /hadoop/hdfs/branches/branch-0.22/src/contrib/hdfsproxy * /hadoop/hdfs/branches/branch-0.22/CHANGES.txt * /hadoop/hdfs/branches/branch-0.22/src/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java * /hadoop/hdfs/branches/branch-0.22/src/c++/libhdfs/configure.ac * /hadoop/hdfs/branches/branch-0.22/src/c++/libhdfs * /hadoop/hdfs/branches/branch-0.22/build.xml Remove AC_TYPE* from the libhdfs Key: HDFS-1619 URL: https://issues.apache.org/jira/browse/HDFS-1619 Project: Hadoop HDFS Issue Type: Improvement Components: libhdfs Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Fix For: 0.22.0 Attachments: HDFS-1619-C99.patch.txt, HDFS-1619.patch.txt, hdfs-1619-2.patch Remove AC_TYPE* from the libhdfs build since we get these via stdint. Currently configure.ac uses AC_TYPE_INT16_T, AC_TYPE_INT32_T, AC_TYPE_INT64_T and AC_TYPE_UINT16_T and thus requires autoconf 2.61 or higher. This prevents using it on such platforms as CentOS/RHEL 5.4 and 5.5. Given that those are pretty popular and also given that it is really difficult to find a platform these days that doesn't natively define intXX_t types I'm curious as to whether we can simply remove those macros or perhaps fail ONLY if we happen to be on such a platform. Here's a link to GNU autoconf docs for your reference: http://www.gnu.org/software/hello/manual/autoconf/Particular-Types.html -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
[ https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045432#comment-13045432 ] Kihwal Lee commented on HDFS-941: - You think 16 a good number for the socket cache (doesn't seem easily chanageable)? If the client's working set size of data nodes in past several seconds is bigger, it means lower locality. If a lot of clients are doing it, each data node is likely to see less data locality, making page cache less effective. This can make more reads cold and the gain from caching connections will start to diminish. Is 16 a good number? IMO, it may actually be too big for typical use cases, but is small enough to not cause trouble. Datanode xceiver protocol should allow reuse of a connection Key: HDFS-941 URL: https://issues.apache.org/jira/browse/HDFS-941 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, hdfs client Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: bc Wong Attachments: HDFS-941-1.patch, HDFS-941-2.patch, HDFS-941-3.patch, HDFS-941-3.patch, HDFS-941-4.patch, HDFS-941-5.patch, HDFS-941-6.patch, HDFS-941-6.patch, HDFS-941-6.patch, hdfs-941.txt, hdfs-941.txt, hdfs941-1.png Right now each connection into the datanode xceiver only processes one operation. In the case that an operation leaves the stream in a well-defined state (eg a client reads to the end of a block successfully) the same connection could be reused for a second operation. This should improve random read performance significantly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-2003: - Attachment: HDFS-2003.diff Fixed the findbugs. I don't think its worth doing it in another JIRA as the changes for it are tiny and don't do very much. I commented the code out instead of removing so that it's clear what was meant to be read. The reason this didn't hit before was that timestamp and atime were being reused in FSEditLogLoader#loadEditRecords(). Separate FSEditLog reading logic from editLog memory state building logic - Key: HDFS-2003 URL: https://issues.apache.org/jira/browse/HDFS-2003 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: Edit log branch (HDFS-1073) Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt Currently FSEditLogLoader has code for reading from an InputStream interleaved with code which updates the FSNameSystem and FSDirectory. This makes it difficult to read an edit log without having a whole load of other object initialised, which is problematic if you want to do things like count how many transactions are in a file etc. This patch separates the reading of the stream and the building of the memory state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045438#comment-13045438 ] Ivan Kelly commented on HDFS-2003: -- Ah, hadn't noticed you'd made a bit of code change in HDFS-2041. Ignore my last patch in that case. Separate FSEditLog reading logic from editLog memory state building logic - Key: HDFS-2003 URL: https://issues.apache.org/jira/browse/HDFS-2003 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: Edit log branch (HDFS-1073) Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt Currently FSEditLogLoader has code for reading from an InputStream interleaved with code which updates the FSNameSystem and FSDirectory. This makes it difficult to read an edit log without having a whole load of other object initialised, which is problematic if you want to do things like count how many transactions are in a file etc. This patch separates the reading of the stream and the building of the memory state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2025) Go Back to File View link is not working in tail.jsp
[ https://issues.apache.org/jira/browse/HDFS-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sravankorumilli updated HDFS-2025: -- Attachment: HDFS-2025_1.patch Updated the patch after addressing the test case failure. Go Back to File View link is not working in tail.jsp Key: HDFS-2025 URL: https://issues.apache.org/jira/browse/HDFS-2025 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.23.0 Reporter: sravankorumilli Assignee: sravankorumilli Priority: Minor Attachments: HDFS-2025.patch, HDFS-2025_1.patch, ScreenShot_1.jpg While browsing the file system. Click on any file link to go to the page where the file contents are displayed, then when we click on '*Tail this file*' link. The control will go to the tail.jsp here when we Click on '*Go Back to File View*' option. HTTP Error page not found will come. This is because the referrer URL is encoded and the encoded URL is itself being used in the '*Go Back to File View*' hyperlink which will be treated as a relative URL and thus the HTTP request will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2025) Go Back to File View link is not working in tail.jsp
[ https://issues.apache.org/jira/browse/HDFS-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sravankorumilli updated HDFS-2025: -- Status: Patch Available (was: Open) Go Back to File View link is not working in tail.jsp Key: HDFS-2025 URL: https://issues.apache.org/jira/browse/HDFS-2025 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.23.0 Reporter: sravankorumilli Assignee: sravankorumilli Priority: Minor Attachments: HDFS-2025.patch, HDFS-2025_1.patch, ScreenShot_1.jpg While browsing the file system. Click on any file link to go to the page where the file contents are displayed, then when we click on '*Tail this file*' link. The control will go to the tail.jsp here when we Click on '*Go Back to File View*' option. HTTP Error page not found will come. This is because the referrer URL is encoded and the encoded URL is itself being used in the '*Go Back to File View*' hyperlink which will be treated as a relative URL and thus the HTTP request will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045470#comment-13045470 ] Hadoop QA commented on HDFS-2003: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481701/HDFS-2003.diff against trunk revision 1132881. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/733//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/733//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/733//console This message is automatically generated. Separate FSEditLog reading logic from editLog memory state building logic - Key: HDFS-2003 URL: https://issues.apache.org/jira/browse/HDFS-2003 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: Edit log branch (HDFS-1073) Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt Currently FSEditLogLoader has code for reading from an InputStream interleaved with code which updates the FSNameSystem and FSDirectory. This makes it difficult to read an edit log without having a whole load of other object initialised, which is problematic if you want to do things like count how many transactions are in a file etc. This patch separates the reading of the stream and the building of the memory state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2025) Go Back to File View link is not working in tail.jsp
[ https://issues.apache.org/jira/browse/HDFS-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045497#comment-13045497 ] Hadoop QA commented on HDFS-2025: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481710/HDFS-2025_1.patch against trunk revision 1132881. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestHFlush +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/734//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/734//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/734//console This message is automatically generated. Go Back to File View link is not working in tail.jsp Key: HDFS-2025 URL: https://issues.apache.org/jira/browse/HDFS-2025 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.23.0 Reporter: sravankorumilli Assignee: sravankorumilli Priority: Minor Attachments: HDFS-2025.patch, HDFS-2025_1.patch, ScreenShot_1.jpg While browsing the file system. Click on any file link to go to the page where the file contents are displayed, then when we click on '*Tail this file*' link. The control will go to the tail.jsp here when we Click on '*Go Back to File View*' option. HTTP Error page not found will come. This is because the referrer URL is encoded and the encoded URL is itself being used in the '*Go Back to File View*' hyperlink which will be treated as a relative URL and thus the HTTP request will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2025) Go Back to File View link is not working in tail.jsp
[ https://issues.apache.org/jira/browse/HDFS-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045509#comment-13045509 ] sravankorumilli commented on HDFS-2025: --- Both the Test Failures are not related. For this change I don't think we require a test case. I verified manually after applying the patch there were no problems in the UI. Go Back to File View link is not working in tail.jsp Key: HDFS-2025 URL: https://issues.apache.org/jira/browse/HDFS-2025 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.23.0 Reporter: sravankorumilli Assignee: sravankorumilli Priority: Minor Attachments: HDFS-2025.patch, HDFS-2025_1.patch, ScreenShot_1.jpg While browsing the file system. Click on any file link to go to the page where the file contents are displayed, then when we click on '*Tail this file*' link. The control will go to the tail.jsp here when we Click on '*Go Back to File View*' option. HTTP Error page not found will come. This is because the referrer URL is encoded and the encoded URL is itself being used in the '*Go Back to File View*' hyperlink which will be treated as a relative URL and thus the HTTP request will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2004) Enable replicating and pinning files to a data node
[ https://issues.apache.org/jira/browse/HDFS-2004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045513#comment-13045513 ] Jason Rutherglen commented on HDFS-2004: I think I mentioned HDFS is probably the wrong place for this, as HDFS should and basically does allow one to implement this functionality today. The end consumer project can easily make the request of the name node etc. Enable replicating and pinning files to a data node --- Key: HDFS-2004 URL: https://issues.apache.org/jira/browse/HDFS-2004 Project: Hadoop HDFS Issue Type: Improvement Components: balancer Affects Versions: 0.23.0 Reporter: Jason Rutherglen Some HDFS applications require that a given file is on the local DataNode. The functionality created here will allow pinning the file to any DataNode. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2040) Only build libhdfs if a flag is passed
[
https://issues.apache.org/jira/browse/HDFS-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045514#comment-13045514
]
Eric Yang commented on HDFS-2040:
-
Is there another patch to remove ${build.platform} from the c++ library build?
Otherwise ${build.platform} should be preserved.
Only build libhdfs if a flag is passed
--
Key: HDFS-2040
URL: https://issues.apache.org/jira/browse/HDFS-2040
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 0.23.0
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
Fix For: 0.23.0
Attachments: hdfs-2040-1.patch
In HDFS-2022 we made ant binary build libhdfs unconditionally, this is a pain
for users who now need to get the native toolchain working to create a
tarball to test a change, and inconsistent with common and MR (see
MAPREDUCE-2559) which only build native code if a flag is passed. Let's
revert to the previous behavior of requiring -Dlibhdfs be passed at build
time. We could also create a new ant target that doesn't build the native
code, however restoring the old behavior seems simplest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2030) Fix the usability of namenode upgrade command
[ https://issues.apache.org/jira/browse/HDFS-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharath Mundlapudi updated HDFS-2030: - Attachment: HDFS-2030-1.patch Attaching the patch Fix the usability of namenode upgrade command - Key: HDFS-2030 URL: https://issues.apache.org/jira/browse/HDFS-2030 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Priority: Minor Fix For: 0.23.0 Attachments: HDFS-2030-1.patch Fixing the Namenode upgrade option along the same line as Namenode format option. If clusterid is not given then clusterid will be automatically generated for the upgrade but if clusterid is given then it will be honored. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1948) Forward port 'hdfs-1520 lightweight namenode operation to trigger lease reccovery'
[ https://issues.apache.org/jira/browse/HDFS-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HDFS-1948: Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Applied patch to TRUNK and to branch-0.22 (HBase won't work on hadoop 0.22.0 if this patch is not present). Thanks for help and review Hairong. Forward port 'hdfs-1520 lightweight namenode operation to trigger lease reccovery' -- Key: HDFS-1948 URL: https://issues.apache.org/jira/browse/HDFS-1948 Project: Hadoop HDFS Issue Type: Task Reporter: stack Assignee: stack Fix For: 0.22.0 Attachments: 1948-part1.txt, 1948-v2.txt, 1948-v3.txt, 1948-v3.txt, 1948-v4-minus_rpc_version_change.txt, 1948-v4.22.txt, 1948-v4.txt This issue is about forward porting from branch-0.20-append the little namenode api that facilitates stealing of a file's lease. The forward port would be an adaption of hdfs-1520 and its companion patches, hdfs-1555 and hdfs-1554, to suit the TRUNK. Intent is to get this fix into 0.22 time willing; i'll run a vote to get ok on getting it added to branch. HBase needs this facility. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2043) TestHFlush failing intermittently
TestHFlush failing intermittently - Key: HDFS-2043 URL: https://issues.apache.org/jira/browse/HDFS-2043 Project: Hadoop HDFS Issue Type: Bug Reporter: Aaron T. Myers I can't reproduce this failure reliably, but it seems like TestHFlush has been failing intermittently, with the frequency increasing of late. Note the following two pre-commit test runs from different JIRAs where TestHFlush seems to have failed spuriously: https://builds.apache.org/job/PreCommit-HDFS-Build/734//testReport/ https://builds.apache.org/job/PreCommit-HDFS-Build/680//testReport/ -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2044) TestQueueProcessingStatistics failing automatic test due to timing issues
TestQueueProcessingStatistics failing automatic test due to timing issues - Key: HDFS-2044 URL: https://issues.apache.org/jira/browse/HDFS-2044 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.20.204.0 Reporter: Matt Foley Assignee: Matt Foley Fix For: 0.20.205.0 The test makes assumptions about timing issues that hold true in workstation environments but not in Hudson auto-test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2044) TestQueueProcessingStatistics failing automatic test due to timing issues
[ https://issues.apache.org/jira/browse/HDFS-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2044: - Affects Version/s: (was: 0.20.204.0) 0.20.203.0 TestQueueProcessingStatistics failing automatic test due to timing issues - Key: HDFS-2044 URL: https://issues.apache.org/jira/browse/HDFS-2044 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.20.203.0 Reporter: Matt Foley Assignee: Matt Foley Fix For: 0.20.205.0 Attachments: IBR_instrumentation_20.203_supl.patch The test makes assumptions about timing issues that hold true in workstation environments but not in Hudson auto-test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2044) TestQueueProcessingStatistics failing automatic test due to timing issues
[ https://issues.apache.org/jira/browse/HDFS-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045596#comment-13045596 ] Suresh Srinivas commented on HDFS-2044: --- +1 for the change TestQueueProcessingStatistics failing automatic test due to timing issues - Key: HDFS-2044 URL: https://issues.apache.org/jira/browse/HDFS-2044 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.20.203.0 Reporter: Matt Foley Assignee: Matt Foley Fix For: 0.20.205.0 Attachments: IBR_instrumentation_20.203_supl.patch The test makes assumptions about timing issues that hold true in workstation environments but not in Hudson auto-test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2044) TestQueueProcessingStatistics failing automatic test due to timing issues
[ https://issues.apache.org/jira/browse/HDFS-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HDFS-2044. -- Resolution: Fixed Note this is a v20 issue only. TestQueueProcessingStatistics failing automatic test due to timing issues - Key: HDFS-2044 URL: https://issues.apache.org/jira/browse/HDFS-2044 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.20.203.0 Reporter: Matt Foley Assignee: Matt Foley Fix For: 0.20.205.0 Attachments: IBR_instrumentation_20.203_supl.patch The test makes assumptions about timing issues that hold true in workstation environments but not in Hudson auto-test. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated HDFS-1954: --- Attachment: HDFS-1954.patch This updated patchremoves the hint, other updates to the patch as we discussed in the comment stream. (Thanks for the FAQ update Konstantin!) Note that this can only be applied to trunk. The reason why the 22 build failed (HDFS-2013) is that the test for this issue TestMissingBlocksAlert is different in 22 vs trunk. In 22 it checks for There are about # while trunk checks for There are # missing blocks. This test passes for me on trunk with this updated patch. I also verified by running the test by hand through the UI interface. Improve corrupt files warning message on NameNode web UI Key: HDFS-1954 URL: https://issues.apache.org/jira/browse/HDFS-1954 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: philo vivero Assignee: Patrick Hunt Fix For: 0.22.0 Attachments: HDFS-1954.patch, HDFS-1954.patch, HDFS-1954.patch Original Estimate: 24h Remaining Estimate: 24h On NameNode web interface, you may get this warning: WARNING : There are about 32 missing blocks. Please check the log or run fsck. If the cluster was started less than 14 days before, it would be great to add: Is dfs.data.dir defined? If at the point of that error message, that parameter could be checked, and error made OMG dfs.data.dir isn't defined! that'd be even better. As is, troubleshooting undefined parameters is a difficult proposition. I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated HDFS-1954: --- Fix Version/s: (was: 0.22.0) Hadoop Flags: (was: [Reviewed]) Status: Patch Available (was: Reopened) Improve corrupt files warning message on NameNode web UI Key: HDFS-1954 URL: https://issues.apache.org/jira/browse/HDFS-1954 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: philo vivero Assignee: Patrick Hunt Attachments: HDFS-1954.patch, HDFS-1954.patch, HDFS-1954.patch Original Estimate: 24h Remaining Estimate: 24h On NameNode web interface, you may get this warning: WARNING : There are about 32 missing blocks. Please check the log or run fsck. If the cluster was started less than 14 days before, it would be great to add: Is dfs.data.dir defined? If at the point of that error message, that parameter could be checked, and error made OMG dfs.data.dir isn't defined! that'd be even better. As is, troubleshooting undefined parameters is a difficult proposition. I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2045) HADOOP_*_HOME environment variables no longer work for tar ball distributions
HADOOP_*_HOME environment variables no longer work for tar ball distributions - Key: HDFS-2045 URL: https://issues.apache.org/jira/browse/HDFS-2045 Project: Hadoop HDFS Issue Type: Bug Reporter: Aaron T. Myers It used to be that you could do the following: # Run `ant bin-package' in your hadoop-common checkout. # Set HADOOP_COMMON_HOME to the built directory of hadoop-common. # Run `ant bin-package' in your hadoop-hdfs checkout. # Set HADOOP_HDFS_HOME to the built directory of hadoop-hdfs. # Set PATH to have HADOOP_HDFS_HOME/bin and HADOOP_COMMON_HOME/bin on it. # Run `hdfs'. \\ \\ As of HDFS-1963, this no longer works since hdfs-config.sh is looking in HADOOP_COMMON_HOME/bin/ for hadoop-config.sh, but it's being placed in HADOOP_COMMON_HOME/libexec. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2014) bin/hdfs no longer works from a source checkout
[ https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045621#comment-13045621 ] Aaron T. Myers commented on HDFS-2014: -- bq. Owen told me verbally in his cube. Add Owen for comment. I've filed HDFS-2045 to discuss this issue. Hopefully Owen can comment there. In the absence of a compelling reason to keep it as-is, I'm inclined to change it back to the previous behavior. bin/hdfs no longer works from a source checkout --- Key: HDFS-2014 URL: https://issues.apache.org/jira/browse/HDFS-2014 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Eric Yang Priority: Critical Fix For: 0.23.0 Attachments: HDFS-2014-1.patch, HDFS-2014-2.patch, HDFS-2014.patch bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a source checkout: todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode ./bin/hdfs: line 22: /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or directory ./bin/hdfs: line 138: cygpath: command not found ./bin/hdfs: line 161: exec: : not found -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2045) HADOOP_*_HOME environment variables no longer work for tar ball distributions
[
https://issues.apache.org/jira/browse/HDFS-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045631#comment-13045631
]
Owen O'Malley commented on HDFS-2045:
-
The HADOOP_*_HOME variables were a bad idea that I take full responsibility for.
The goal should be to move the deployment and development environments closer
together rather than have two completely different structures.
Currently, you can do:
{code}
cd common
ant rpm
rpm -i build/hadoop-common-*.rpm
cd ../hdfs
ant rpm
rpm -i build/hadoop-hdfs-*.rpm
{code}
which is far easier to understand and you are running it like it is deployed.
Of course, you can also deploy using the tarballs instead of rpm or debs.
Do you have ideas for making the dev easier? I assume part of Nigel's goal of
moving the subversion trees to gether is to eventually have a shared build
directory.
HADOOP_*_HOME environment variables no longer work for tar ball distributions
-
Key: HDFS-2045
URL: https://issues.apache.org/jira/browse/HDFS-2045
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Aaron T. Myers
It used to be that you could do the following:
# Run `ant bin-package' in your hadoop-common checkout.
# Set HADOOP_COMMON_HOME to the built directory of hadoop-common.
# Run `ant bin-package' in your hadoop-hdfs checkout.
# Set HADOOP_HDFS_HOME to the built directory of hadoop-hdfs.
# Set PATH to have HADOOP_HDFS_HOME/bin and HADOOP_COMMON_HOME/bin on it.
# Run `hdfs'.
\\
\\
As of HDFS-1963, this no longer works since hdfs-config.sh is looking in
HADOOP_COMMON_HOME/bin/ for hadoop-config.sh, but it's being placed in
HADOOP_COMMON_HOME/libexec.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2045) HADOOP_*_HOME environment variables no longer work for tar ball distributions
[ https://issues.apache.org/jira/browse/HDFS-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045633#comment-13045633 ] Owen O'Malley commented on HDFS-2045: - I'm sorry, I read too fast. Are you just proposing to fix the hdfs-config.sh? HADOOP_*_HOME environment variables no longer work for tar ball distributions - Key: HDFS-2045 URL: https://issues.apache.org/jira/browse/HDFS-2045 Project: Hadoop HDFS Issue Type: Bug Reporter: Aaron T. Myers It used to be that you could do the following: # Run `ant bin-package' in your hadoop-common checkout. # Set HADOOP_COMMON_HOME to the built directory of hadoop-common. # Run `ant bin-package' in your hadoop-hdfs checkout. # Set HADOOP_HDFS_HOME to the built directory of hadoop-hdfs. # Set PATH to have HADOOP_HDFS_HOME/bin and HADOOP_COMMON_HOME/bin on it. # Run `hdfs'. \\ \\ As of HDFS-1963, this no longer works since hdfs-config.sh is looking in HADOOP_COMMON_HOME/bin/ for hadoop-config.sh, but it's being placed in HADOOP_COMMON_HOME/libexec. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045636#comment-13045636 ] Hadoop QA commented on HDFS-1954: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481740/HDFS-1954.patch against trunk revision 1133114. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/735//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/735//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/735//console This message is automatically generated. Improve corrupt files warning message on NameNode web UI Key: HDFS-1954 URL: https://issues.apache.org/jira/browse/HDFS-1954 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: philo vivero Assignee: Patrick Hunt Attachments: HDFS-1954.patch, HDFS-1954.patch, HDFS-1954.patch Original Estimate: 24h Remaining Estimate: 24h On NameNode web interface, you may get this warning: WARNING : There are about 32 missing blocks. Please check the log or run fsck. If the cluster was started less than 14 days before, it would be great to add: Is dfs.data.dir defined? If at the point of that error message, that parameter could be checked, and error made OMG dfs.data.dir isn't defined! that'd be even better. As is, troubleshooting undefined parameters is a difficult proposition. I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2014) bin/hdfs no longer works from a source checkout
[ https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045642#comment-13045642 ] Owen O'Malley commented on HDFS-2014: - Sorry about that. On the other hand, having different layouts for the different deployment vehicles is pretty broken too. bin/hdfs no longer works from a source checkout --- Key: HDFS-2014 URL: https://issues.apache.org/jira/browse/HDFS-2014 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Eric Yang Priority: Critical Fix For: 0.23.0 Attachments: HDFS-2014-1.patch, HDFS-2014-2.patch, HDFS-2014.patch bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a source checkout: todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode ./bin/hdfs: line 22: /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or directory ./bin/hdfs: line 138: cygpath: command not found ./bin/hdfs: line 161: exec: : not found -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2045) HADOOP_*_HOME environment variables no longer work for tar ball distributions
[ https://issues.apache.org/jira/browse/HDFS-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045648#comment-13045648 ] Eric Yang commented on HDFS-2045: - The state of current code in trunk. Developers: # Direct source tree execution, specify HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME works. # Source tarball, same as 1. Ops: # Binary tarball, decompress all binary tarball to the same destination without prefix. There is no need to use environment variable. # RPM/DEB works as described in Owen's comment, no need to specify environment variables. HADOOP_*_HOME environment variables no longer work for tar ball distributions - Key: HDFS-2045 URL: https://issues.apache.org/jira/browse/HDFS-2045 Project: Hadoop HDFS Issue Type: Bug Reporter: Aaron T. Myers It used to be that you could do the following: # Run `ant bin-package' in your hadoop-common checkout. # Set HADOOP_COMMON_HOME to the built directory of hadoop-common. # Run `ant bin-package' in your hadoop-hdfs checkout. # Set HADOOP_HDFS_HOME to the built directory of hadoop-hdfs. # Set PATH to have HADOOP_HDFS_HOME/bin and HADOOP_COMMON_HOME/bin on it. # Run `hdfs'. \\ \\ As of HDFS-1963, this no longer works since hdfs-config.sh is looking in HADOOP_COMMON_HOME/bin/ for hadoop-config.sh, but it's being placed in HADOOP_COMMON_HOME/libexec. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2046) Force entropy to come from non-true random for tests
Force entropy to come from non-true random for tests Key: HDFS-2046 URL: https://issues.apache.org/jira/browse/HDFS-2046 Project: Hadoop HDFS Issue Type: Improvement Components: build, test Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Same as HADOOP-7335 but for HDFS -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045657#comment-13045657 ] Patrick Hunt commented on HDFS-1954: TestHDFSCLI is currently broken in trunk, not caused by this patch: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-trunk/690/testReport/junit/org.apache.hadoop.cli/TestHDFSCLI/ also TestMissingBlocksAlert is covering this change. Improve corrupt files warning message on NameNode web UI Key: HDFS-1954 URL: https://issues.apache.org/jira/browse/HDFS-1954 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: philo vivero Assignee: Patrick Hunt Attachments: HDFS-1954.patch, HDFS-1954.patch, HDFS-1954.patch Original Estimate: 24h Remaining Estimate: 24h On NameNode web interface, you may get this warning: WARNING : There are about 32 missing blocks. Please check the log or run fsck. If the cluster was started less than 14 days before, it would be great to add: Is dfs.data.dir defined? If at the point of that error message, that parameter could be checked, and error made OMG dfs.data.dir isn't defined! that'd be even better. As is, troubleshooting undefined parameters is a difficult proposition. I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2046) Force entropy to come from non-true random for tests
[ https://issues.apache.org/jira/browse/HDFS-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2046: -- Attachment: hdfs-2046.txt Force entropy to come from non-true random for tests Key: HDFS-2046 URL: https://issues.apache.org/jira/browse/HDFS-2046 Project: Hadoop HDFS Issue Type: Improvement Components: build, test Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: hdfs-2046.txt Same as HADOOP-7335 but for HDFS -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2046) Force entropy to come from non-true random for tests
[ https://issues.apache.org/jira/browse/HDFS-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2046: -- Status: Patch Available (was: Open) Force entropy to come from non-true random for tests Key: HDFS-2046 URL: https://issues.apache.org/jira/browse/HDFS-2046 Project: Hadoop HDFS Issue Type: Improvement Components: build, test Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: hdfs-2046.txt Same as HADOOP-7335 but for HDFS -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1875) MiniDFSCluster hard-codes dfs.datanode.address to localhost
[ https://issues.apache.org/jira/browse/HDFS-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-1875: - Resolution: Fixed Status: Resolved (was: Patch Available) +1. The one auto-test failure is unrelated. Committed to trunk. Thanks, Eric! MiniDFSCluster hard-codes dfs.datanode.address to localhost --- Key: HDFS-1875 URL: https://issues.apache.org/jira/browse/HDFS-1875 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 0.23.0 Attachments: HDFS-1875.patch, HDFS-1875.patch When creating RPC addresses that represent the communication sockets for each simulated DataNode, the MiniDFSCluster class hard-codes the address of the dfs.datanode.address port to be 127.0.0.1:0 The DataNodeCluster test tool uses the MiniDFSCluster class to create a selected number of simulated datanodes on a single host. In the DataNodeCluster setup, the NameNode is not simulated but is started as a separate daemon. The problem is that if the write requrests into the simulated datanodes are originated on a host that is not the same host running the simulated datanodes, the connections are refused. This is because the RPC sockets that are started by MiniDFSCluster are for localhost (127.0.0.1) and are not accessible from outside that same machine. It is proposed that the MiniDFSCluster.setupDatanodeAddress() method be overloaded in order to accommodate an environment where the NameNode is on one host, the client is on another host, and the simulated DataNodes are on yet another host (or even multiple hosts simulating multiple DataNodes each). The overloaded API would add a parameter that would be used as the basis for creating the RPS sockets. By default, it would remain 127.0.0.1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2016) 1073: remove/archive unneeded/old storage files
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2016: -- Attachment: hdfs-2016.txt Rebased on current branch. I'm going to commit this version. Feel free to post-commit review if there are further comments. 1073: remove/archive unneeded/old storage files --- Key: HDFS-2016 URL: https://issues.apache.org/jira/browse/HDFS-2016 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) Attachments: hdfs-2016.txt, hdfs-2016.txt, hdfs-2016.txt This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and archives them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2016) 1073: remove/archive unneeded/old storage files
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-2016. --- Resolution: Fixed 1073: remove/archive unneeded/old storage files --- Key: HDFS-2016 URL: https://issues.apache.org/jira/browse/HDFS-2016 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) Attachments: hdfs-2016.txt, hdfs-2016.txt, hdfs-2016.txt This JIRA is for the infrastructure that periodically scans the storage directories for files that are no longer necessary and archives them. The default archival is just to delete them, but we may implement something fancier in the future (eg move to SAN or HDFS) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2047) Improve TestNamespace and TestEditLog in 1073 branch
Improve TestNamespace and TestEditLog in 1073 branch Key: HDFS-2047 URL: https://issues.apache.org/jira/browse/HDFS-2047 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Todd Lipcon Assignee: Todd Lipcon -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2047) Improve TestNamespace and TestEditLog in 1073 branch
[ https://issues.apache.org/jira/browse/HDFS-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2047: -- Component/s: test Description: These tests currently have some test cases that don't make sense after HDFS-1073. This JIRA is to update these tests to do the equivalent things on 1073. Affects Version/s: Edit log branch (HDFS-1073) Fix Version/s: Edit log branch (HDFS-1073) Improve TestNamespace and TestEditLog in 1073 branch Key: HDFS-2047 URL: https://issues.apache.org/jira/browse/HDFS-2047 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) These tests currently have some test cases that don't make sense after HDFS-1073. This JIRA is to update these tests to do the equivalent things on 1073. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2047) Improve TestNamespace and TestEditLog in 1073 branch
[ https://issues.apache.org/jira/browse/HDFS-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2047: -- Attachment: hdfs-2047.txt Enumerated changes: - FSImage.saveFSImageInAllDirs wasn't throwing an exception even if there were no non-failed storage dirs - TestEditLog now makes sure that the formatted NN has an fsimage_0 file - TestEditLog now makes sure that, if the NN starts with any unfinalized logs (eg after a crash) it will save a new checkpoint on startup - Removed old fault points from TestSaveNamespace that no longer made sense (eg moveCurrent) and added new faults at various points in the image saving process. Improve TestNamespace and TestEditLog in 1073 branch Key: HDFS-2047 URL: https://issues.apache.org/jira/browse/HDFS-2047 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) Attachments: hdfs-2047.txt These tests currently have some test cases that don't make sense after HDFS-1073. This JIRA is to update these tests to do the equivalent things on 1073. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2045) HADOOP_*_HOME environment variables no longer work for tar ball distributions
[ https://issues.apache.org/jira/browse/HDFS-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045688#comment-13045688 ] Aaron T. Myers commented on HDFS-2045: -- bq. I'm sorry, I read too fast. Are you just proposing to fix the hdfs-config.sh? Unfortunately, I think it would take more than that to restore the previous behavior. Since the jars, etc are now located based solely on the single HADOOP_PREFIX env var, one cannot have separate distribution directories for HDFS, MR, and Common. My main issue is that all of this rearranging/reworking of the tarball and development layouts were done under the auspices of adding RPM support. I'll be honest and say that I didn't review the RPM work hardly at all, because I assumed from the title that it would not affect tarballs or the dev environment. Had there been separate JIRAs along the lines of rearrange the layout of distribution builds and add RPM support, I likely would have reviewed the former. HADOOP_*_HOME environment variables no longer work for tar ball distributions - Key: HDFS-2045 URL: https://issues.apache.org/jira/browse/HDFS-2045 Project: Hadoop HDFS Issue Type: Bug Reporter: Aaron T. Myers It used to be that you could do the following: # Run `ant bin-package' in your hadoop-common checkout. # Set HADOOP_COMMON_HOME to the built directory of hadoop-common. # Run `ant bin-package' in your hadoop-hdfs checkout. # Set HADOOP_HDFS_HOME to the built directory of hadoop-hdfs. # Set PATH to have HADOOP_HDFS_HOME/bin and HADOOP_COMMON_HOME/bin on it. # Run `hdfs'. \\ \\ As of HDFS-1963, this no longer works since hdfs-config.sh is looking in HADOOP_COMMON_HOME/bin/ for hadoop-config.sh, but it's being placed in HADOOP_COMMON_HOME/libexec. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2045) HADOOP_*_HOME environment variables no longer work for tar ball distributions
[ https://issues.apache.org/jira/browse/HDFS-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045694#comment-13045694 ] Eric Yang commented on HDFS-2045: - HADOOP_COMMON_HOME, HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME were results of splitting the source code into three different submodules. While this works fine for developer to isolate each project, it makes configuration difficult for production use. HDFS and MAPRED run as their own uid. The amount of configuration just multiples. To solve this problem, there are a couple options: Option 1. Modify jar file which contains all common shell script in common jar file, when binary tarball is built, the common shell scripts are rearranged submerged into the binary tarball distribution, and completely remove HADOOP_*_HOME environment variables. $HADOOP_PREFIX is the only hint (generated from shell script path, no need to define in the environment) to all hadoop programs where the bits are exactly layout. When HDFS or MAPREDUCE is deployed, there is no need to deploy COMMON tarball. To make this work for developers, *-config.sh should be moved to $HADOOP_PREFIX/libexec. During the build process, hadoop-common-*.jar is extract for common shell scripts. Both developer and binary layout are closer to each other. (When project is converted to maven, this keeps hdfs/mapreduce loosely coupled and reduce duplicated shell scripts.) Option 2. Preserve HADOOP_*_HOME for source code execution. Environment driven layout does not work on binary tarball. Change the prefix tarball from hadoop-[common|mapred|hdfs]-0.23.0-SNAPSHOT to hadoop-[version] for easy extraction. Option 3. Enable HADOOP_*_HOME for binary tarball. (Risk of crashing the system due to bad environment variable setup) Option 4. Merge hdfs/mapreduce back to the same project, but create as subdirectories to reduce duplicated shell scripts. I am incline to vote for option 2. HADOOP_*_HOME environment variables no longer work for tar ball distributions - Key: HDFS-2045 URL: https://issues.apache.org/jira/browse/HDFS-2045 Project: Hadoop HDFS Issue Type: Bug Reporter: Aaron T. Myers It used to be that you could do the following: # Run `ant bin-package' in your hadoop-common checkout. # Set HADOOP_COMMON_HOME to the built directory of hadoop-common. # Run `ant bin-package' in your hadoop-hdfs checkout. # Set HADOOP_HDFS_HOME to the built directory of hadoop-hdfs. # Set PATH to have HADOOP_HDFS_HOME/bin and HADOOP_COMMON_HOME/bin on it. # Run `hdfs'. \\ \\ As of HDFS-1963, this no longer works since hdfs-config.sh is looking in HADOOP_COMMON_HOME/bin/ for hadoop-config.sh, but it's being placed in HADOOP_COMMON_HOME/libexec. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2046) Force entropy to come from non-true random for tests
[ https://issues.apache.org/jira/browse/HDFS-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045699#comment-13045699 ] Hadoop QA commented on HDFS-2046: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481752/hdfs-2046.txt against trunk revision 1133114. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/736//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/736//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/736//console This message is automatically generated. Force entropy to come from non-true random for tests Key: HDFS-2046 URL: https://issues.apache.org/jira/browse/HDFS-2046 Project: Hadoop HDFS Issue Type: Improvement Components: build, test Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: hdfs-2046.txt Same as HADOOP-7335 but for HDFS -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2048) 1073: Improve upgrade tests from 0.22
1073: Improve upgrade tests from 0.22 - Key: HDFS-2048 URL: https://issues.apache.org/jira/browse/HDFS-2048 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Todd Lipcon Assignee: Todd Lipcon -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2048) 1073: Improve upgrade tests from 0.22
[ https://issues.apache.org/jira/browse/HDFS-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2048: -- Description: TestDFSUpgradeFromImage currently tests an upgrade from 0.22, but doesn't test that the image checksum field is properly respected during the upgrade. This JIRA is to improve those tests by also testing the case where the image has been corrupted. Affects Version/s: Edit log branch (HDFS-1073) Fix Version/s: Edit log branch (HDFS-1073) 1073: Improve upgrade tests from 0.22 - Key: HDFS-2048 URL: https://issues.apache.org/jira/browse/HDFS-2048 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) TestDFSUpgradeFromImage currently tests an upgrade from 0.22, but doesn't test that the image checksum field is properly respected during the upgrade. This JIRA is to improve those tests by also testing the case where the image has been corrupted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2048) 1073: Improve upgrade tests from 0.22
[ https://issues.apache.org/jira/browse/HDFS-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2048: -- Attachment: hdfs-2048.txt Patch includes a new test case which tests upgrade from the 0.22 image, but before doing so, corrupts the md5 sum stored in the version file. This also fixes a bug in the upgrade path so that this test case passes. 1073: Improve upgrade tests from 0.22 - Key: HDFS-2048 URL: https://issues.apache.org/jira/browse/HDFS-2048 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) Attachments: hdfs-2048.txt TestDFSUpgradeFromImage currently tests an upgrade from 0.22, but doesn't test that the image checksum field is properly respected during the upgrade. This JIRA is to improve those tests by also testing the case where the image has been corrupted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045714#comment-13045714 ] Todd Lipcon commented on HDFS-2003: --- For reviewers, please review http://issues.apache.org/jira/secure/attachment/12481642/hdfs-2003.txt No tests included since this is a straight refactor. The findbugs warnings are fixed by HDFS-2041. Separate FSEditLog reading logic from editLog memory state building logic - Key: HDFS-2003 URL: https://issues.apache.org/jira/browse/HDFS-2003 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: Edit log branch (HDFS-1073) Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt Currently FSEditLogLoader has code for reading from an InputStream interleaved with code which updates the FSNameSystem and FSDirectory. This makes it difficult to read an edit log without having a whole load of other object initialised, which is problematic if you want to do things like count how many transactions are in a file etc. This patch separates the reading of the stream and the building of the memory state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2040) Only build libhdfs if a flag is passed
[
https://issues.apache.org/jira/browse/HDFS-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045720#comment-13045720
]
Eli Collins commented on HDFS-2040:
---
bq. Is there another patch to remove ${build.platform} from the c++ library
build?
Nope.
bq. Otherwise ${build.platform} should be preserved.
Why? c++/lib is a link to c++/${build.platform}/lib, the idea for this link is
to not have to worry about the build.platform in all the places we want to
point to the c++ lib.
Only build libhdfs if a flag is passed
--
Key: HDFS-2040
URL: https://issues.apache.org/jira/browse/HDFS-2040
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 0.23.0
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
Fix For: 0.23.0
Attachments: hdfs-2040-1.patch
In HDFS-2022 we made ant binary build libhdfs unconditionally, this is a pain
for users who now need to get the native toolchain working to create a
tarball to test a change, and inconsistent with common and MR (see
MAPREDUCE-2559) which only build native code if a flag is passed. Let's
revert to the previous behavior of requiring -Dlibhdfs be passed at build
time. We could also create a new ant target that doesn't build the native
code, however restoring the old behavior seems simplest.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[
https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045721#comment-13045721
]
Todd Lipcon commented on HDFS-2003:
---
Did a self-review by going through the diff with the old code on the left and
the new Op classes on the right. Two small notes, one of which is a real bug:
- AddCloseOp has the following problem: in the old code, it would call
fsNamesys.adjustReplication(), and then the adjusted replication would be
passed to the readBlocks call. In the new code, the *unadjusted* replication is
passed. So, the blocks end up with the wrong replication count.
- We should rename {{SetGenstampOp.lw}} to something more meaningful (perhaps
{{genStamp}})
Separate FSEditLog reading logic from editLog memory state building logic
-
Key: HDFS-2003
URL: https://issues.apache.org/jira/browse/HDFS-2003
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: Edit log branch (HDFS-1073)
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Fix For: Edit log branch (HDFS-1073)
Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff,
HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt
Currently FSEditLogLoader has code for reading from an InputStream
interleaved with code which updates the FSNameSystem and FSDirectory. This
makes it difficult to read an edit log without having a whole load of other
object initialised, which is problematic if you want to do things like count
how many transactions are in a file etc.
This patch separates the reading of the stream and the building of the memory
state.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2040) Only build libhdfs if a flag is passed
[ https://issues.apache.org/jira/browse/HDFS-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2040: -- Attachment: hdfs-2040-2.patch Updated patch attached that doesn't use the symlink. Not a big deal either way to me, this patch is more consistent with MR-2559. Only build libhdfs if a flag is passed -- Key: HDFS-2040 URL: https://issues.apache.org/jira/browse/HDFS-2040 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Fix For: 0.23.0 Attachments: hdfs-2040-1.patch, hdfs-2040-2.patch In HDFS-2022 we made ant binary build libhdfs unconditionally, this is a pain for users who now need to get the native toolchain working to create a tarball to test a change, and inconsistent with common and MR (see MAPREDUCE-2559) which only build native code if a flag is passed. Let's revert to the previous behavior of requiring -Dlibhdfs be passed at build time. We could also create a new ant target that doesn't build the native code, however restoring the old behavior seems simplest. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2040) Only build libhdfs if a flag is passed
[ https://issues.apache.org/jira/browse/HDFS-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045728#comment-13045728 ] Todd Lipcon commented on HDFS-2040: --- +1 Only build libhdfs if a flag is passed -- Key: HDFS-2040 URL: https://issues.apache.org/jira/browse/HDFS-2040 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Fix For: 0.23.0 Attachments: hdfs-2040-1.patch, hdfs-2040-2.patch In HDFS-2022 we made ant binary build libhdfs unconditionally, this is a pain for users who now need to get the native toolchain working to create a tarball to test a change, and inconsistent with common and MR (see MAPREDUCE-2559) which only build native code if a flag is passed. Let's revert to the previous behavior of requiring -Dlibhdfs be passed at build time. We could also create a new ant target that doesn't build the native code, however restoring the old behavior seems simplest. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[
https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-2003:
--
Attachment: hdfs-2003.txt
It turns out that the bug mentioned above does not actually cause a problem. I
added a unit test here to be sure of it -- the BlockInfo objects do get created
with a replication count that's too low, but before the {{triplets}} array is
used, {{ensureCapacity}} will resize it appropriately.
I think this patch is ready for commit if someone can please review.
Separate FSEditLog reading logic from editLog memory state building logic
-
Key: HDFS-2003
URL: https://issues.apache.org/jira/browse/HDFS-2003
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: Edit log branch (HDFS-1073)
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Fix For: Edit log branch (HDFS-1073)
Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff,
HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt,
hdfs-2003.txt
Currently FSEditLogLoader has code for reading from an InputStream
interleaved with code which updates the FSNameSystem and FSDirectory. This
makes it difficult to read an edit log without having a whole load of other
object initialised, which is problematic if you want to do things like count
how many transactions are in a file etc.
This patch separates the reading of the stream and the building of the memory
state.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2040) Only build libhdfs if a flag is passed
[ https://issues.apache.org/jira/browse/HDFS-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2040: -- Hadoop Flags: [Reviewed] Status: Patch Available (was: Open) Only build libhdfs if a flag is passed -- Key: HDFS-2040 URL: https://issues.apache.org/jira/browse/HDFS-2040 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Fix For: 0.23.0 Attachments: hdfs-2040-1.patch, hdfs-2040-2.patch In HDFS-2022 we made ant binary build libhdfs unconditionally, this is a pain for users who now need to get the native toolchain working to create a tarball to test a change, and inconsistent with common and MR (see MAPREDUCE-2559) which only build native code if a flag is passed. Let's revert to the previous behavior of requiring -Dlibhdfs be passed at build time. We could also create a new ant target that doesn't build the native code, however restoring the old behavior seems simplest. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2040) Only build libhdfs if a flag is passed
[ https://issues.apache.org/jira/browse/HDFS-2040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045735#comment-13045735 ] Hadoop QA commented on HDFS-2040: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481763/hdfs-2040-2.patch against trunk revision 1133225. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/737//console This message is automatically generated. Only build libhdfs if a flag is passed -- Key: HDFS-2040 URL: https://issues.apache.org/jira/browse/HDFS-2040 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Eli Collins Assignee: Eli Collins Priority: Minor Fix For: 0.23.0 Attachments: hdfs-2040-1.patch, hdfs-2040-2.patch In HDFS-2022 we made ant binary build libhdfs unconditionally, this is a pain for users who now need to get the native toolchain working to create a tarball to test a change, and inconsistent with common and MR (see MAPREDUCE-2559) which only build native code if a flag is passed. Let's revert to the previous behavior of requiring -Dlibhdfs be passed at build time. We could also create a new ant target that doesn't build the native code, however restoring the old behavior seems simplest. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13045754#comment-13045754 ] Hadoop QA commented on HDFS-2003: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481764/hdfs-2003.txt against trunk revision 1133225. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/738//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/738//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/738//console This message is automatically generated. Separate FSEditLog reading logic from editLog memory state building logic - Key: HDFS-2003 URL: https://issues.apache.org/jira/browse/HDFS-2003 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: Edit log branch (HDFS-1073) Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: 2003-delta.txt, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, HDFS-2003.diff, hdfs-2003.txt, hdfs-2003.txt, hdfs-2003.txt Currently FSEditLogLoader has code for reading from an InputStream interleaved with code which updates the FSNameSystem and FSDirectory. This makes it difficult to read an edit log without having a whole load of other object initialised, which is problematic if you want to do things like count how many transactions are in a file etc. This patch separates the reading of the stream and the building of the memory state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2049) Should add the column name tips for HDFS shell command
Should add the column name tips for HDFS shell command -- Key: HDFS-2049 URL: https://issues.apache.org/jira/browse/HDFS-2049 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.21.0 Reporter: Denny Ye :abc:root bin/hadoop fs -count -q /ABC 11/06/07 18:05:54 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=30 11/06/07 18:05:54 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id none infnone inf 13 372 0 hdfs://abc:9000/ABC -- I got the result columns without column name. It make me confused. Actually, it should like this in a kindly matter. :abc:root bin/hadoop fs -count -q /ABC 11/06/07 18:05:54 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=30 11/06/07 18:05:54 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id QUOTA, REMAINING_QUATA, SPACE_QUOTA, REMAINING_SPACE_QUOTA, DIR_COUNT, FILE_COUNT, CONTENT_SIZE, FILE_NAME none infnone inf 13 372 0 hdfs://abc:9000/ABC -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2049) Should add the column name tips for HDFS shell command
[ https://issues.apache.org/jira/browse/HDFS-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-2049: --- Hadoop Flags: [Incompatible change] Should add the column name tips for HDFS shell command -- Key: HDFS-2049 URL: https://issues.apache.org/jira/browse/HDFS-2049 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.21.0 Reporter: Denny Ye Labels: comand, shell :abc:root bin/hadoop fs -count -q /ABC 11/06/07 18:05:54 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=30 11/06/07 18:05:54 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id none infnone inf 13 372 0 hdfs://abc:9000/ABC -- I got the result columns without column name. It make me confused. Actually, it should like this in a kindly matter. :abc:root bin/hadoop fs -count -q /ABC 11/06/07 18:05:54 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=30 11/06/07 18:05:54 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id QUOTA, REMAINING_QUATA, SPACE_QUOTA, REMAINING_SPACE_QUOTA, DIR_COUNT, FILE_COUNT, CONTENT_SIZE, FILE_NAME none infnone inf 13 372 0 hdfs://abc:9000/ABC -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
