[jira] [Updated] (HDFS-2459) Separate datatypes for Journal protocol
[ https://issues.apache.org/jira/browse/HDFS-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2459: -- Attachment: HDFS-2459.txt Separate datatypes for Journal protocol --- Key: HDFS-2459 URL: https://issues.apache.org/jira/browse/HDFS-2459 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-2459.txt, HDFS-2459.txt This jira separates for JournalProtocol the wire types from the types used by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2459) Separate datatypes for Journal protocol
[ https://issues.apache.org/jira/browse/HDFS-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129623#comment-13129623 ] Hadoop QA commented on HDFS-2459: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499516/HDFS-2459.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1384//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1384//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1384//console This message is automatically generated. Separate datatypes for Journal protocol --- Key: HDFS-2459 URL: https://issues.apache.org/jira/browse/HDFS-2459 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-2459.txt, HDFS-2459.txt This jira separates for JournalProtocol the wire types from the types used by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2188) HDFS-1580: Make FSEditLog create its journals from a list of URIs rather than NNStorage
[ https://issues.apache.org/jira/browse/HDFS-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129715#comment-13129715 ] Hudson commented on HDFS-2188: -- Integrated in Hadoop-Mapreduce-trunk #864 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/864/]) HDFS-2188. Make FSEditLog create its journals from a list of URIs rather than NNStorage. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185354 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSImageTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java HDFS-1580: Make FSEditLog create its journals from a list of URIs rather than NNStorage --- Key: HDFS-2188 URL: https://issues.apache.org/jira/browse/HDFS-2188 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 0.24.0 Attachments: HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff Currently, FSEditLog retrieves the list of Journals to create from NNStorage. Obviously this is file specific. This JIRA aims to remove this restriction to make it possible to create journals of custom types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2188) HDFS-1580: Make FSEditLog create its journals from a list of URIs rather than NNStorage
[ https://issues.apache.org/jira/browse/HDFS-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129733#comment-13129733 ] Hudson commented on HDFS-2188: -- Integrated in Hadoop-Hdfs-trunk #834 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/834/]) HDFS-2188. Make FSEditLog create its journals from a list of URIs rather than NNStorage. Contributed by Ivan Kelly. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185354 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSImageTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java HDFS-1580: Make FSEditLog create its journals from a list of URIs rather than NNStorage --- Key: HDFS-2188 URL: https://issues.apache.org/jira/browse/HDFS-2188 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 0.24.0 Attachments: HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff Currently, FSEditLog retrieves the list of Journals to create from NNStorage. Obviously this is file specific. This JIRA aims to remove this restriction to make it possible to create journals of custom types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129792#comment-13129792 ] Uma Maheswara Rao G commented on HDFS-2467: --- Looks duplicate of HDFS-2449 ( findbugs caught this) HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2453) tail using a webhdfs uri throws an error
[ https://issues.apache.org/jira/browse/HDFS-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2453: - Attachment: h2453_20111018.patch h2453_20111018_0.20s.patch h2453_20111018_0.20s.patch h2453_20111018.patch Fixed TestByteRangeInputStream. tail using a webhdfs uri throws an error Key: HDFS-2453 URL: https://issues.apache.org/jira/browse/HDFS-2453 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Attachments: h2453_20111017.patch, h2453_20111017_0.20s.patch, h2453_20111017b.patch, h2453_20111017b_0.20s.patch, h2453_20111018.patch, h2453_20111018_0.20s.patch /usr//bin/hadoop --config /etc/hadoop dfs -tail webhdfs://NN:50070/file tail: HTTP_PARTIAL expected, received 200 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1447) Make getGenerationStampFromFile() more efficient, so it doesn't reprocess full directory listing for every block
[ https://issues.apache.org/jira/browse/HDFS-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129829#comment-13129829 ] Todd Lipcon commented on HDFS-1447: --- Also, what percent of startup time is devoted to CPU usage of this scan? In HDFS-2384 I uploaded a C program which does the block scan as efficiently as possible - but most of the gains there are from sorting by inum before statting. Make getGenerationStampFromFile() more efficient, so it doesn't reprocess full directory listing for every block Key: HDFS-1447 URL: https://issues.apache.org/jira/browse/HDFS-1447 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Affects Versions: 0.20.2 Reporter: Matt Foley Assignee: Matt Foley Attachments: HDFS-1447.patch, Test_HDFS_1447_NotForCommitt.java.patch Make getGenerationStampFromFile() more efficient. Currently this routine is called by addToReplicasMap() for every blockfile in the directory tree, and it walks each file's containing directory on every call. There is a simple refactoring that should make it more efficient. This work item is one of four sub-tasks for HDFS-1443, Improve Datanode startup time. The fix will probably be folded into sibling task HDFS-1446, which is already refactoring the method that calls getGenerationStampFromFile(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2461) Support HDFS file name globbing in libhdfs
[
https://issues.apache.org/jira/browse/HDFS-2461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129852#comment-13129852
]
Mariappan Asokan commented on HDFS-2461:
I thought more on this. Since the Java globStatus() method already queried the
name node to retrieve the status information, for the sake of efficiency I
think we can change the function signature. Also, conforming to already
existing hdfsListDirectory(), I decided to return an array of structures rather
than array of pointers. This will enable reusing the existing C function
hdfsFreeFileInfo(). I also added the path filter function in the interface.
Filtering will be done in the C implementation. Following is the description
of the prototype of the single function:
{code:title=hdfs.h}
/**
* Path filter function prototype.
* @param pathName path name passed to this function.
* @return 0 if the path name has to be excluded; a non-zero otherwise.
*/
typedef int (*PathFilter)(const char * pathName);
/**
* hdfsGlobStatus - Get status for all HDFS file names that match a glob
* pattern. The returned result will be an array of hdfsFileInfo structures.
* The array is sorted by file names.
* The function hdfsFreeFileInfo() should be called to free this array and its
* contents.
* @param fs The configured filesystem handle.
* @param globPattern The glob pattern(as supported by Hadoop implementation) to
* match file names against.
* @param filter A path filter function. If this is NULL, no filtering will be
* done after glob expansion.
* @param numEntries pointer to an integer in which the number of entries in the
* returned array will be returned. This will be set to -1 in case of error.
* @return Returns a dynamically-allocated array of hdfsFileInfo structures; if
* there is no match or an error, a NULL value will be returned. An error
* condition can be identified by testing numEntries.
*/
hdfsFileInfo * hdfsGlobStatus(hdfsFS fs, const char *globPattern,
PathFilter filter, int *numEntries);
{code}
If anyone has any comments, please let me know.
Thanks.
Support HDFS file name globbing in libhdfs
--
Key: HDFS-2461
URL: https://issues.apache.org/jira/browse/HDFS-2461
Project: Hadoop HDFS
Issue Type: Improvement
Components: libhdfs
Reporter: Mariappan Asokan
Priority: Minor
This is to enhance the C API in libhdfs to support HDFS file name globbing.
The proposal is to keep the new API simple and return a list of matched HDFS
path names. Callers can use existing hdfsGetPathInfo() to get additional
information on each of the matched path. Following code snippet shows the
proposed API enhancements:
{code:title=hdfs.h}
/**
* hdfsGlob - Get all the HDFS file names that match a glob pattern. The
* returned result will be sorted by the file names. The last element in the
* array is NULL. The function hdfsFreeGlob() should be called to free this
* array and its contents.
* @param fs The configured filesystem handle.
* @param globPattern The glob pattern to match file names against. Note that
* this is not a POSIX regular expression but rather a POSIX glob pattern.
* @return Returns a dynamically-allocated array of strings; if there is no
* match, an array with one entry that has a NULL value will be returned. If
* there is an error, NULL will be returned.
*/
char ** hdfsGlob(hdfsFS fs, const char *globPattern);
/**
* hdfsFreeGlob - Free up the array returned by hdfsGlob().
* @param globResult The array of dynamically-allocated strings returned by
* hdfsGlob().
*/
void hdfsFreeGlob(char **globResult);
{code}
Please comment on the above proposed API. I will start the implementation
and testing. However, I need a committer to work with.
Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2468) webhdfs create a file and send permission does not set the permission to the expected value
webhdfs create a file and send permission does not set the permission to the expected value --- Key: HDFS-2468 URL: https://issues.apache.org/jira/browse/HDFS-2468 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Issue a create call and send permission(666). The file that is created gets the permission rw-r--r-- instead of expected rw-rw-rw- -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2469) webhdfs create a file and send buffersize=0 hangs the request
webhdfs create a file and send buffersize=0 hangs the request - Key: HDFS-2469 URL: https://issues.apache.org/jira/browse/HDFS-2469 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Issue a create call and send buffersize=0 the user gets the redirect and upon sending the request to the new location the request does not return with the content. Should we set a min and max value for buffersize? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2469) webhdfs create a file and send buffersize=0 hangs the request
[ https://issues.apache.org/jira/browse/HDFS-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129873#comment-13129873 ] Arpit Gupta commented on HDFS-2469: --- what i also noticed is that when this call was made the cpu resources being used start growing on the namenode. we should prob set min and max values. webhdfs create a file and send buffersize=0 hangs the request - Key: HDFS-2469 URL: https://issues.apache.org/jira/browse/HDFS-2469 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Issue a create call and send buffersize=0 the user gets the redirect and upon sending the request to the new location the request does not return with the content. Should we set a min and max value for buffersize? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HDFS-2467: Attachment: h-2467.patch This fixes the equality and adds a test. HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129876#comment-13129876 ] Konstantin Shvachko commented on HDFS-2452: --- Sleeping an arbitrary # of seconds is not a good solution, as it makes tests flaky, passing on some nodes and failing on other because of different environments. If you need to wait, then wait for a specific event (thread started in the case) rather than relying on passing time. DataXceiver thread already handled the Throwable there. You are right, mocking getXceiverCount() doesn't work. May be Cos can help with that. OutOfMemoryError in DataXceiverServer takes down the DataNode - Key: HDFS-2452 URL: https://issues.apache.org/jira/browse/HDFS-2452 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Uma Maheswara Rao G Fix For: 0.22.0 Attachments: HDFS-2452-22branch.1.patch, HDFS-2452-22branch.patch OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2449) Fix the findbugs introduced in HftpFileSystem
[
https://issues.apache.org/jira/browse/HDFS-2449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley updated HDFS-2449:
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Sorry for the duplication, but I've got a better fix and test over in HDFS-2467.
Fix the findbugs introduced in HftpFileSystem
-
Key: HDFS-2449
URL: https://issues.apache.org/jira/browse/HDFS-2449
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Attachments: HDFS-2449.patch
Call to equals() comparing different types
{code}
if (t.getService().toString().equals(hdfsServiceName)) {
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2453) tail using a webhdfs uri throws an error
[ https://issues.apache.org/jira/browse/HDFS-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129880#comment-13129880 ] Hadoop QA commented on HDFS-2453: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499554/h2453_20111018.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1385//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1385//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1385//console This message is automatically generated. tail using a webhdfs uri throws an error Key: HDFS-2453 URL: https://issues.apache.org/jira/browse/HDFS-2453 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Attachments: h2453_20111017.patch, h2453_20111017_0.20s.patch, h2453_20111017b.patch, h2453_20111017b_0.20s.patch, h2453_20111018.patch, h2453_20111018_0.20s.patch /usr//bin/hadoop --config /etc/hadoop dfs -tail webhdfs://NN:50070/file tail: HTTP_PARTIAL expected, received 200 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HDFS-2467: Status: Patch Available (was: Open) HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129886#comment-13129886 ] Hadoop QA commented on HDFS-2467: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499565/h-2467.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1386//console This message is automatically generated. HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2468) webhdfs create a file and send permission does not set the permission to the expected value
[ https://issues.apache.org/jira/browse/HDFS-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Gupta updated HDFS-2468: -- Description: Issue a create call using rest api and send permission(666). The file that is created gets the permission rw-r--r-- instead of expected rw-rw-rw- (was: Issue a create call and send permission(666). The file that is created gets the permission rw-r--r-- instead of expected rw-rw-rw-) webhdfs create a file and send permission does not set the permission to the expected value --- Key: HDFS-2468 URL: https://issues.apache.org/jira/browse/HDFS-2468 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Issue a create call using rest api and send permission(666). The file that is created gets the permission rw-r--r-- instead of expected rw-rw-rw- -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129910#comment-13129910 ] Jitendra Nath Pandey commented on HDFS-2467: Changes to .gitignore and TestHftpFileSystem.java look unintentional. The patch looks ok otherwise, +1. HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2119) hdfs version should print out information similar to what hadoop version prints out
[ https://issues.apache.org/jira/browse/HDFS-2119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129909#comment-13129909 ] Mahadev konar commented on HDFS-2119: - @ravi, Can we not do this using shared version for all the 3 common/hdfs/mapred. I need to look into what the new tar assembly in hadoop-dist module is, but I think we could probably just calculate the version somewhere common and use it in hdfs/common/mapred. Will update after looking into the the poms. hdfs version should print out information similar to what hadoop version prints out --- Key: HDFS-2119 URL: https://issues.apache.org/jira/browse/HDFS-2119 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 0.23.0 Attachments: HDFS-2119.1.patch, HDFS-2119.2.patch, HDFS-2119.3.patch Implement version in hdfs CLI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2463) webhdfs able to write data to a file that was readonly
[ https://issues.apache.org/jira/browse/HDFS-2463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Gupta resolved HDFS-2463. --- Resolution: Invalid resolving as invalid based on Nicholas's comments webhdfs able to write data to a file that was readonly -- Key: HDFS-2463 URL: https://issues.apache.org/jira/browse/HDFS-2463 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2462) webhdfs create a file that exists and overwrite it also changes the permission of the file
[ https://issues.apache.org/jira/browse/HDFS-2462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Gupta resolved HDFS-2462. --- Resolution: Invalid resolving as invalid based on Nicholas's comments webhdfs create a file that exists and overwrite it also changes the permission of the file -- Key: HDFS-2462 URL: https://issues.apache.org/jira/browse/HDFS-2462 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HDFS-2467: Status: Open (was: Patch Available) HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HDFS-2467: Status: Patch Available (was: Open) HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1447) Make getGenerationStampFromFile() more efficient, so it doesn't reprocess full directory listing for every block
[ https://issues.apache.org/jira/browse/HDFS-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129925#comment-13129925 ] Uma Maheswara Rao G commented on HDFS-1447: --- @Suresh: Thanks for taking a look. Yes, i know the max files per dir is 64 by default. Please check other JIRAs in this umberilla for further optimizations. Wanted to use the same scan for recoverTmpUnlinkedFiles with other JIRA. With the less file we may not get good improvements and with this single JIRA. Above figures are to show the relative improvement in optimizing the loops. @Todd, Yes, the scanning logic was good in HDFS-2384. But in FSDataSet case, we need to maintain the tree as well rite. Also in C code i have seen there is a genstamps array with blkids. Long array will consume more memory right? Make getGenerationStampFromFile() more efficient, so it doesn't reprocess full directory listing for every block Key: HDFS-1447 URL: https://issues.apache.org/jira/browse/HDFS-1447 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Affects Versions: 0.20.2 Reporter: Matt Foley Assignee: Matt Foley Attachments: HDFS-1447.patch, Test_HDFS_1447_NotForCommitt.java.patch Make getGenerationStampFromFile() more efficient. Currently this routine is called by addToReplicasMap() for every blockfile in the directory tree, and it walks each file's containing directory on every call. There is a simple refactoring that should make it more efficient. This work item is one of four sub-tasks for HDFS-1443, Improve Datanode startup time. The fix will probably be folded into sibling task HDFS-1446, which is already refactoring the method that calls getGenerationStampFromFile(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HDFS-2467: Attachment: h-2467.patch Removed the extra changes, although we should fix the .gitignore. HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2469) webhdfs create a file and send buffersize=0 hangs the request
[ https://issues.apache.org/jira/browse/HDFS-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129930#comment-13129930 ] Arpit Gupta commented on HDFS-2469: --- could we also add a check for the block size when a value that is not a multiple of io.bytes.per.checksum is sent the user is redirected and then the call fails. webhdfs create a file and send buffersize=0 hangs the request - Key: HDFS-2469 URL: https://issues.apache.org/jira/browse/HDFS-2469 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Issue a create call and send buffersize=0 the user gets the redirect and upon sending the request to the new location the request does not return with the content. Should we set a min and max value for buffersize? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
[ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129938#comment-13129938 ] Alejandro Abdelnur commented on HDFS-2178: -- I've forgot to mention that another use case a generic http proxy won't cover is that Hoop can be used to HTTP proxy any Hadoop FileSystem implementation over HTTP. Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities -- Key: HDFS-2178 URL: https://issues.apache.org/jira/browse/HDFS-2178 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 0.23.0 Attachments: HDFSoverHTTP-API.html, HdfsHttpAPI.pdf We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) for HDFS Proxy. Hoop provides access to all Hadoop Distributed File System (HDFS) operations (read and write) over HTTP/S. The Hoop server component is a REST HTTP gateway to HDFS supporting all file system operations. It can be accessed using standard HTTP tools (i.e. curl and wget), HTTP libraries from different programing languages (i.e. Perl, Java Script) as well as using the Hoop client. The Hoop server component is a standard Java web-application and it has been implemented using Jersey (JAX-RS). The Hoop client component is an implementation of Hadoop FileSystem client that allows using the familiar Hadoop filesystem API to access HDFS data through a Hoop server. Repo: https://github.com/cloudera/hoop Docs: http://cloudera.github.com/hoop Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/ Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for Kerberos HTTP SPNEGO authentication). To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be done first, as well as the Alfredo contribution (HADOOP-7119). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2453) tail using a webhdfs uri throws an error
[ https://issues.apache.org/jira/browse/HDFS-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2453: - Attachment: h2453_20111018_TestByteRangeInputStream_0.20s.patch h2453_20111018_TestByteRangeInputStream_0.20s.patch: added TestByteRangeInputStream to 0.20s tail using a webhdfs uri throws an error Key: HDFS-2453 URL: https://issues.apache.org/jira/browse/HDFS-2453 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Attachments: h2453_20111017.patch, h2453_20111017_0.20s.patch, h2453_20111017b.patch, h2453_20111017b_0.20s.patch, h2453_20111018.patch, h2453_20111018_0.20s.patch, h2453_20111018_TestByteRangeInputStream_0.20s.patch /usr//bin/hadoop --config /etc/hadoop dfs -tail webhdfs://NN:50070/file tail: HTTP_PARTIAL expected, received 200 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2453) tail using a webhdfs uri throws an error
[ https://issues.apache.org/jira/browse/HDFS-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129959#comment-13129959 ] Hadoop QA commented on HDFS-2453: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499583/h2453_20111018_TestByteRangeInputStream_0.20s.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1388//console This message is automatically generated. tail using a webhdfs uri throws an error Key: HDFS-2453 URL: https://issues.apache.org/jira/browse/HDFS-2453 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Attachments: h2453_20111017.patch, h2453_20111017_0.20s.patch, h2453_20111017b.patch, h2453_20111017b_0.20s.patch, h2453_20111018.patch, h2453_20111018_0.20s.patch, h2453_20111018_TestByteRangeInputStream_0.20s.patch /usr//bin/hadoop --config /etc/hadoop dfs -tail webhdfs://NN:50070/file tail: HTTP_PARTIAL expected, received 200 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2469) webhdfs create a file and send buffersize=0 hangs the request
[ https://issues.apache.org/jira/browse/HDFS-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129982#comment-13129982 ] Arpit Gupta commented on HDFS-2469: --- another parameter 'replication' if the user sends a value of 0 can we reject the request at namenode rather than after the redirect webhdfs create a file and send buffersize=0 hangs the request - Key: HDFS-2469 URL: https://issues.apache.org/jira/browse/HDFS-2469 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Arpit Gupta Assignee: Tsz Wo (Nicholas), SZE Issue a create call and send buffersize=0 the user gets the redirect and upon sending the request to the new location the request does not return with the content. Should we set a min and max value for buffersize? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13129992#comment-13129992 ] Hadoop QA commented on HDFS-2467: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499577/h-2467.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.TestRestartDFS org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool org.apache.hadoop.hdfs.TestPipelines org.apache.hadoop.hdfs.TestBlocksScheduledCounter +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1387//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1387//console This message is automatically generated. HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130009#comment-13130009 ] Hudson commented on HDFS-2467: -- Integrated in Hadoop-Hdfs-trunk-Commit #1180 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1180/]) HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation tokens. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185842 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130011#comment-13130011 ] Hudson commented on HDFS-2467: -- Integrated in Hadoop-Common-trunk-Commit #1101 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1101/]) HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation tokens. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185842 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-2452: - Status: Open (was: Patch Available) We aren't running auto-validation of .22 patches anyway. OutOfMemoryError in DataXceiverServer takes down the DataNode - Key: HDFS-2452 URL: https://issues.apache.org/jira/browse/HDFS-2452 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Uma Maheswara Rao G Fix For: 0.22.0 Attachments: HDFS-2452-22branch.1.patch, HDFS-2452-22branch.patch OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130018#comment-13130018 ] Hudson commented on HDFS-2467: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1120 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1120/]) HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation tokens. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185842 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1971) HA: Send block report from datanode to both active and standby namenodes
[ https://issues.apache.org/jira/browse/HDFS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130037#comment-13130037 ] Sanjay Radia commented on HDFS-1971: I have attached a partial patch. * I have created a BPServiceToStandby class as a subclass of the BPOfferService class. However, I think I need to revisit this. The reason is that when NN failover occurs the thread that sends the BRs to standby should perform the same function as the main BPOfferService thread; we don't want to start a new BPOfferService since it will have to reregister and send new block reports. Hence I believe that I am thinking of simply adding a state to BPOfferService. Will post an updated patch. * The block-additions-deletions are passed from the primary BP thread to the secondary BP thread. * The patch generates the BRs again. The code can be easily modified to optimize for option 1 or option 3. ** @eli - double-scanning DN disks ... Note for BRs we do not scan the disks but instead generate the BR from the Block-map data structure. HA: Send block report from datanode to both active and standby namenodes Key: HDFS-1971 URL: https://issues.apache.org/jira/browse/HDFS-1971 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, name-node Reporter: Suresh Srinivas Assignee: Sanjay Radia Attachments: DualBlockReports.pdf, daulBr1.patch To enable hot standby namenode, the standby node must have current information for - namenode state (image + edits) and block location information. This jira addresses keeping the block location information current in the standby node. To do this, the proposed solution is to send block reports from the datanodes to both the active and the standby namenode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated HDFS-2467: Resolution: Fixed Target Version/s: 0.23.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this to trunk and 0.23. HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130077#comment-13130077 ] Hudson commented on HDFS-2467: -- Integrated in Hadoop-Hdfs-0.23-Commit #19 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/19/]) HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation tokens. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185897 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130079#comment-13130079 ] Hudson commented on HDFS-2467: -- Integrated in Hadoop-Common-0.23-Commit #18 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/18/]) HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation tokens. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185897 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2467) HftpFileSystem uses incorrect compare for finding delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130086#comment-13130086 ] Hudson commented on HDFS-2467: -- Integrated in Hadoop-Mapreduce-0.23-Commit #19 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/19/]) HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation tokens. (omalley) omalley : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1185897 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java HftpFileSystem uses incorrect compare for finding delegation tokens --- Key: HDFS-2467 URL: https://issues.apache.org/jira/browse/HDFS-2467 Project: Hadoop HDFS Issue Type: Bug Reporter: Owen O'Malley Assignee: Owen O'Malley Fix For: 0.23.0 Attachments: h-2467.patch, h-2467.patch When looking for hdfs delegation tokens, Hftp converts the service to a string and compares it to a text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[
https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-2450:
--
Attachment: HDFS-2450.patch
I added more definite checks to {{FileSystem.checkPath(Path)}} to properly
detect equivalence between long and short hostnames.
Tweak {{DistributedFileSystem}} to stop unnecessarily mangling the given URIs
which led to the problem being noticed. This should prevent the DFS from not
realizing it owns its paths, and not make the aforementioned {{checkPath}}
changes have to work very hard.
Commit tests passed, will post full test results tomorrow morning.
Only complete hostname is supported to access data via hdfs://
--
Key: HDFS-2450
URL: https://issues.apache.org/jira/browse/HDFS-2450
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.20.205.0
Reporter: Rajit Saha
Assignee: Daryn Sharp
Attachments: HDFS-2450.patch
If my complete hostname is host1.abc.xyz.com, only complete hostname must be
used to access data via hdfs://
I am running following in .20.205 Client to get data from .20.205 NN (host1)
$hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp
copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com
Usage: java FsShell [-copyFromLocal localsrc ... dst]
$hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/
copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
hdfs://host1.abc.xyz.com
Usage: java FsShell [-copyFromLocal localsrc ... dst]
$hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/
copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
hdfs://host1.abc.xyz.com
Usage: java FsShell [-copyFromLocal localsrc ... dst]
Only following is supported
$hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[ https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-2450: -- Status: Patch Available (was: Open) Only complete hostname is supported to access data via hdfs:// -- Key: HDFS-2450 URL: https://issues.apache.org/jira/browse/HDFS-2450 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Rajit Saha Assignee: Daryn Sharp Attachments: HDFS-2450.patch If my complete hostname is host1.abc.xyz.com, only complete hostname must be used to access data via hdfs:// I am running following in .20.205 Client to get data from .20.205 NN (host1) $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com Usage: java FsShell [-copyFromLocal localsrc ... dst] $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/ copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: hdfs://host1.abc.xyz.com Usage: java FsShell [-copyFromLocal localsrc ... dst] $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/ copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: hdfs://host1.abc.xyz.com Usage: java FsShell [-copyFromLocal localsrc ... dst] Only following is supported $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2447) Distcp with hdfs:// passed with error in JT log while copying from .20.204 to .20.205 ( with useIp=false)
[ https://issues.apache.org/jira/browse/HDFS-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp reassigned HDFS-2447: - Assignee: Daryn Sharp Distcp with hdfs:// passed with error in JT log while copying from .20.204 to .20.205 ( with useIp=false) -- Key: HDFS-2447 URL: https://issues.apache.org/jira/browse/HDFS-2447 Project: Hadoop HDFS Issue Type: Bug Components: security Affects Versions: 0.20.205.0 Reporter: Rajit Saha Assignee: Daryn Sharp I tried to copy file from .20.204 to .20.205 by distcp over hdfs:// while using hadoop.security.token.service.use_ip=false in core-site.xml. The copy was successful but found error org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: exception in .20.205 JT. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[ https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130131#comment-13130131 ] Hadoop QA commented on HDFS-2450: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12499612/HDFS-2450.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1389//console This message is automatically generated. Only complete hostname is supported to access data via hdfs:// -- Key: HDFS-2450 URL: https://issues.apache.org/jira/browse/HDFS-2450 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Rajit Saha Assignee: Daryn Sharp Attachments: HDFS-2450.patch If my complete hostname is host1.abc.xyz.com, only complete hostname must be used to access data via hdfs:// I am running following in .20.205 Client to get data from .20.205 NN (host1) $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com Usage: java FsShell [-copyFromLocal localsrc ... dst] $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/ copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: hdfs://host1.abc.xyz.com Usage: java FsShell [-copyFromLocal localsrc ... dst] $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/ copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: hdfs://host1.abc.xyz.com Usage: java FsShell [-copyFromLocal localsrc ... dst] Only following is supported $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1971) HA: Send block report from datanode to both active and standby namenodes
[ https://issues.apache.org/jira/browse/HDFS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130156#comment-13130156 ] Todd Lipcon commented on HDFS-1971: --- Hi Sanjay. It looks like the base of your diff might be the wrong branch - it includes a bunch of changes that aren't related to this JIRA. Can you reformat the diff? HA: Send block report from datanode to both active and standby namenodes Key: HDFS-1971 URL: https://issues.apache.org/jira/browse/HDFS-1971 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, name-node Reporter: Suresh Srinivas Assignee: Sanjay Radia Attachments: DualBlockReports.pdf, daulBr1.patch To enable hot standby namenode, the standby node must have current information for - namenode state (image + edits) and block location information. This jira addresses keeping the block location information current in the standby node. To do this, the proposed solution is to send block reports from the datanodes to both the active and the standby namenode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-140) When a file is deleted, its blocks remain in the blocksmap till the next block report from Datanode
[ https://issues.apache.org/jira/browse/HDFS-140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-140: Target Version/s: 0.20.206.0 Fix Version/s: (was: 0.20.205.0) When a file is deleted, its blocks remain in the blocksmap till the next block report from Datanode --- Key: HDFS-140 URL: https://issues.apache.org/jira/browse/HDFS-140 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.1 Reporter: dhruba borthakur Assignee: Uma Maheswara Rao G Attachments: HDFS-140.20security205.patch When a file is deleted, the namenode sends out block deletions messages to the appropriate datanodes. However, the namenode does not delete these blocks from the blocksmap. Instead, the processing of the next block report from the datanode causes these blocks to get removed from the blocksmap. If we desire to make block report processing less frequent, this issue needs to be addressed. Also, this introduces indeterministic behaviout to a a few unit tests. Another factor to consider is to ensure that duplicate block detection is not compromised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1122) client block verification may result in blocks in DataBlockScanner prematurely
[ https://issues.apache.org/jira/browse/HDFS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-1122: - Target Version/s: 0.20-append, 0.20.205.0 Fix Version/s: (was: 0.20-append) If no one chooses to port this patch to 0.20-append, perhaps we should go ahead and close this jira. client block verification may result in blocks in DataBlockScanner prematurely -- Key: HDFS-1122 URL: https://issues.apache.org/jira/browse/HDFS-1122 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20-append, 0.20.205.0 Reporter: sam rash Assignee: sam rash Fix For: 0.20.205.0 Attachments: HDFS-1122-20-security.2.patch, hdfs-1122-for-0.20.txt found that when the DN uses client verification of a block that is open for writing, it will add it to the DataBlockScanner prematurely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2300) TestFileAppend4 and TestMultiThreadedSync fail on 20.append and 20-security.
[ https://issues.apache.org/jira/browse/HDFS-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2300: - Target Version/s: 0.20-append, 0.20.205.0 Fix Version/s: (was: 0.20-append) If no one chooses to port this patch to 0.20-append, perhaps we should go ahead and close this jira. TestFileAppend4 and TestMultiThreadedSync fail on 20.append and 20-security. Key: HDFS-2300 URL: https://issues.apache.org/jira/browse/HDFS-2300 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20-append Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.20.205.0 Attachments: HDFS-2300.20-append.1.patch TestFileAppend4 and TestMultiThreadedSync fail on the 20.append and 20-security branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2385) Support delegation token renewal in webhdfs
[ https://issues.apache.org/jira/browse/HDFS-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2385: - Target Version/s: 0.20.205.0, 0.23.0 Support delegation token renewal in webhdfs --- Key: HDFS-2385 URL: https://issues.apache.org/jira/browse/HDFS-2385 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.20.205.0 Attachments: h2358_20111001_0.20s.patch, h2385_20111004_0.20s.patch, h2385_20111004b_0.20s.patch, h2385_20111004b_amendment2_0.20s.patch, h2385_20111004b_amendment_0.20s.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2451) TestNodeCount.testNodeCount failes with NPE
[
https://issues.apache.org/jira/browse/HDFS-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HDFS-2451:
-
Attachment: HDFS-2451.patch
Seems like the call to {{blockManager.countNodes()}} is happening under
namesystem's write locking elsewhere. Otherwise a race condition is likely.
TestNodeCount.testNodeCount failes with NPE
---
Key: HDFS-2451
URL: https://issues.apache.org/jira/browse/HDFS-2451
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.22.0
Reporter: Konstantin Boudnik
Priority: Blocker
Attachments: HDFS-2451.patch
in the [commit build #97|http://is.gd/UPuXg2] TestNodeCount.testNodeCount
failed with NPE
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.BlockManager.countNodes(BlockManager.java:1436)
at
org.apache.hadoop.hdfs.server.namenode.TestNodeCount.__CLR2_4_39bdgm6whp(TestNodeCount.java:119)
at
org.apache.hadoop.hdfs.server.namenode.TestNodeCount.testNodeCount(TestNodeCount.java:40)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
[ https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1975: --- Attachment: HDFS-1975-HA.patch This is a very early version of the patch. In this patch I am letting the datanode call to block if generation stamp is not up to date. I will upload a patch very soon which won't block the call and instead queue the message. The patch is not review ready yet, but indicates the approach being taken. HA: Support for sharing the namenode state from active to standby. -- Key: HDFS-1975 URL: https://issues.apache.org/jira/browse/HDFS-1975 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Suresh Srinivas Assignee: Jitendra Nath Pandey Attachments: HDFS-1975-HA.patch, hdfs-1975.txt, hdfs-1975.txt To enable hot standby namenode, the standby node must have current information for - namenode state (image + edits) and block location information. This jira addresses keeping the namenode state current in the standby node. To do this, the proposed solution in this jira is to use a shared storage to store the namenode state. Note one could also build an alternative solution by augmenting the backup node. A seperate jira could explore this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2470) NN should automatically set permissions on dfs.namenode.*.dir
NN should automatically set permissions on dfs.namenode.*.dir - Key: HDFS-2470 URL: https://issues.apache.org/jira/browse/HDFS-2470 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.24.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Much as the DN currently sets the correct permissions for the dfs.datanode.data.dir, the NN should do the same for the dfs.namenode.(name|edit).dir. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[
https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HDFS-2452:
-
Attachment: HDFS-2452-22branch.patch
With respect to the case I think getting rid of Mockito.doThrow and spying on
{{DataXceiverServer}} should do the trick.
So, something along the lines
{code}
...
DataNode dn = Mockito.mock(DataNode.class);
DataXceiverServer server = new DataXceiverServer(sock, new Configuration(),
dn);
DataXceiverServer spyServer = spy(server);
doThrow(new OutOfMemoryError(Faulting...)).when(spyServer).run();
dn.shouldRun = true;
...
{code}
would work, I believe. Fixed patch version is attached.
OutOfMemoryError in DataXceiverServer takes down the DataNode
-
Key: HDFS-2452
URL: https://issues.apache.org/jira/browse/HDFS-2452
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Uma Maheswara Rao G
Fix For: 0.22.0
Attachments: HDFS-2452-22branch.1.patch, HDFS-2452-22branch.patch,
HDFS-2452-22branch.patch
OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn
a new data transfer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130323#comment-13130323 ] Konstantin Boudnik commented on HDFS-2452: -- Did you see [my comment| https://issues.apache.org/jira/browse/HDFS-2452?focusedCommentId=13130311page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13130311] and the new patch? OutOfMemoryError in DataXceiverServer takes down the DataNode - Key: HDFS-2452 URL: https://issues.apache.org/jira/browse/HDFS-2452 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Uma Maheswara Rao G Fix For: 0.22.0 Attachments: HDFS-2452-22branch.1.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2452) OutOfMemoryError in DataXceiverServer takes down the DataNode
[ https://issues.apache.org/jira/browse/HDFS-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130333#comment-13130333 ] Konstantin Boudnik commented on HDFS-2452: -- Right, I agree - there's no way in the current implementation to mock this internal instance of DataXceiver. I see how it can be done using fault injection with ease, but I am not sure if you guys want to consider such approach. OutOfMemoryError in DataXceiverServer takes down the DataNode - Key: HDFS-2452 URL: https://issues.apache.org/jira/browse/HDFS-2452 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Uma Maheswara Rao G Fix For: 0.22.0 Attachments: HDFS-2452-22branch.1.patch, HDFS-2452-22branch.patch, HDFS-2452-22branch.patch OutOfMemoryError brings down DataNode, when DataXceiverServer tries to spawn a new data transfer thread. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
