[jira] [Created] (HADOOP-12351) Can't run test-patch with start-build-env.sh
Jakob Homan created HADOOP-12351: Summary: Can't run test-patch with start-build-env.sh Key: HADOOP-12351 URL: https://issues.apache.org/jira/browse/HADOOP-12351 Project: Hadoop Common Issue Type: Bug Components: test, yetus Reporter: Jakob Homan Priority: Minor The Docker instance started by start-build-env.sh drops the user into a root directory wherein other directories (say ~/download) are not accessible, so one cannot pull patches to test into it. The dev-support/test-patch.sh script requires a clean environment with no extra files, such as the patch to test. Between these two restrictions, one can't use the docker env to run test-patch. We should either -v a volume where patches can be stashed, or allow test-patch.sh to ignore patch files' existences. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-12068) Remove @author tags
[ https://issues.apache.org/jira/browse/HADOOP-12068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-12068. -- Resolution: Invalid Wrong project. > Remove @author tags > --- > > Key: HADOOP-12068 > URL: https://issues.apache.org/jira/browse/HADOOP-12068 > Project: Hadoop Common > Issue Type: Bug > Reporter: Jakob Homan > > Apache generally frowns on @author tags (as they imply ownership of the code > by a single individual, see > https://blogs.oracle.com/ahe/entry/coding_conventions_and_attribution [best > asf link is not working, annoyingly] and > https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute as an > example) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12068) Remove @author tags
Jakob Homan created HADOOP-12068: Summary: Remove @author tags Key: HADOOP-12068 URL: https://issues.apache.org/jira/browse/HADOOP-12068 Project: Hadoop Common Issue Type: Bug Reporter: Jakob Homan Apache generally frowns on @author tags (as they imply ownership of the code by a single individual, see https://blogs.oracle.com/ahe/entry/coding_conventions_and_attribution [best asf link is not working, annoyingly] and https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute as an example) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12016) Typo in FileSystem:: listStatusIterator
Jakob Homan created HADOOP-12016:
Summary: Typo in FileSystem:: listStatusIterator
Key: HADOOP-12016
URL: https://issues.apache.org/jira/browse/HADOOP-12016
Project: Hadoop Common
Issue Type: Bug
Components: fs
Reporter: Jakob Homan
Priority: Trivial
{code} public RemoteIterator listStatusIterator(final Path p)
throws FileNotFoundException, IOException {
return new RemoteIterator() {
//...
throw new NoSuchElementException("No more entry in " + p);
}
return stats[i++];
}{code}
Should be 'no more entries'
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HADOOP-12009) FileSystemContractBaseTest:testListStatus should not assume listStatus returns sorted results
Jakob Homan created HADOOP-12009:
Summary: FileSystemContractBaseTest:testListStatus should not
assume listStatus returns sorted results
Key: HADOOP-12009
URL: https://issues.apache.org/jira/browse/HADOOP-12009
Project: Hadoop Common
Issue Type: Improvement
Components: test
Reporter: Jakob Homan
Priority: Minor
FileSystem.listStatus does not guarantee that implementations will return
sorted entries:
{code} /**
* List the statuses of the files/directories in the given path if the path is
* a directory.
*
* @param f given path
* @return the statuses of the files/directories in the given patch
* @throws FileNotFoundException when the path does not exist;
* IOException see specific implementation
*/
public abstract FileStatus[] listStatus(Path f) throws FileNotFoundException,
IOException;{code}
However, FileSystemContractBaseTest, expects the elements to come back sorted:
{code}Path[] testDirs = { path("/test/hadoop/a"),
path("/test/hadoop/b"),
path("/test/hadoop/c/1"), };
// ...
paths = fs.listStatus(path("/test/hadoop"));
assertEquals(3, paths.length);
assertEquals(path("/test/hadoop/a"), paths[0].getPath());
assertEquals(path("/test/hadoop/b"), paths[1].getPath());
assertEquals(path("/test/hadoop/c"), paths[2].getPath());{code}
We should pass this test as long as all the paths are there, regardless of
their ordering.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [VOTE] Change by-laws on release votes: 5 days instead of 7
+1 (binding) On Tue, Jun 24, 2014 at 10:33 AM, Zhijie Shen wrote: > +1 (non-binding) > > > On Wed, Jun 25, 2014 at 1:26 AM, Aaron T. Myers wrote: > > > +1 (binding) > > > > -- > > Aaron T. Myers > > Software Engineer, Cloudera > > > > > > On Tue, Jun 24, 2014 at 1:53 AM, Arun C Murthy > > wrote: > > > > > Folks, > > > > > > As discussed, I'd like to call a vote on changing our by-laws to > change > > > release votes from 7 days to 5. > > > > > > I've attached the change to by-laws I'm proposing. > > > > > > Please vote, the vote will the usual period of 7 days. > > > > > > thanks, > > > Arun > > > > > > > > > > > > [main]$ svn diff > > > Index: author/src/documentation/content/xdocs/bylaws.xml > > > === > > > --- author/src/documentation/content/xdocs/bylaws.xml (revision > > 1605015) > > > +++ author/src/documentation/content/xdocs/bylaws.xml (working copy) > > > @@ -344,7 +344,16 @@ > > > Votes are open for a period of 7 days to allow all active > > > voters time to consider the vote. Votes relating to code > > > changes are not subject to a strict timetable but should be > > > -made as timely as possible. > > > +made as timely as possible. > > > + > > > + > > > + Product Release - Vote Timeframe > > > + Release votes, alone, run for a period of 5 days. All > > other > > > + votes are subject to the above timeframe of 7 days. > > > + > > > + > > > + > > > + > > > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > > > -- > Zhijie Shen > Hortonworks Inc. > http://hortonworks.com/ > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
HADOOP JIRA near full of contributors, error message not helpful
FYI, ran into trouble adding a new contributor in the HADOOP common project (https://issues.apache.org/jira/browse/INFRA-7293). Looks like we've hit the limit for contributors, but the error message is not very helpful. Suresh has cleaned up some, but would be good to keep in mind in the future when adding people. -Jakob
[jira] [Resolved] (HADOOP-8322) Log entry for successful auth has misspelling
[
https://issues.apache.org/jira/browse/HADOOP-8322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan resolved HADOOP-8322.
-
Resolution: Duplicate
> Log entry for successful auth has misspelling
> -
>
> Key: HADOOP-8322
> URL: https://issues.apache.org/jira/browse/HADOOP-8322
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Jakob Homan
>Assignee: Boris Shkolnik
>Priority: Minor
> Labels: newbie
>
> Server.java:
> {code} private static final String AUTH_SUCCESSFULL_FOR = "Auth successfull
> for "; {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8322) Log entry for successful auth has misspelling
Jakob Homan created HADOOP-8322:
---
Summary: Log entry for successful auth has misspelling
Key: HADOOP-8322
URL: https://issues.apache.org/jira/browse/HADOOP-8322
Project: Hadoop Common
Issue Type: Improvement
Reporter: Jakob Homan
Priority: Minor
Server.java:
{code} private static final String AUTH_SUCCESSFULL_FOR = "Auth successfull
for "; {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8229) DistCp doesn't handle non-existent paths correctly
DistCp doesn't handle non-existent paths correctly
--
Key: HADOOP-8229
URL: https://issues.apache.org/jira/browse/HADOOP-8229
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1.0.0, 0.20.2
Reporter: Jakob Homan
Assume /user/jhoman/blork doesn't exist:
{noformat}[tardis]$ hadoop distcp 'hftp://nn:50070/user/jhoman/blork' /tmp/plork
12/03/29 22:04:33 INFO tools.DistCp:
srcPaths=[hftp://nn:50070/user/jhoman/blork]
12/03/29 22:04:33 INFO tools.DistCp: destPath=/tmp/plork
[Fatal Error] :1:173: XML document structures must start and end within the
same entity.
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.io.IOException: invalid xml directory content
at
org.apache.hadoop.hdfs.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:427)
at
org.apache.hadoop.hdfs.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:432)
at
org.apache.hadoop.hdfs.HftpFileSystem.getFileStatus(HftpFileSystem.java:461)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
at org.apache.hadoop.tools.DistCp.checkSrcPath(DistCp.java:636)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:656)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
Caused by: org.xml.sax.SAXParseException: XML document structures must start
and end within the same entity.
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1231)
at
org.apache.hadoop.hdfs.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:421)
... 9 more{noformat}
This is because the ListPathsServlet hits an NPE when it calls
nnproxy.getFileInfo(path); on the non-existent path and just bails, leaving the
resulting XML unformed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7927) Can't build packages 205+ on OSX
Can't build packages 205+ on OSX
Key: HADOOP-7927
URL: https://issues.apache.org/jira/browse/HADOOP-7927
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 0.20.205.0, 1.0.0
Reporter: Jakob Homan
Currently the ant build script tries to reference the native directories, which
are not built on OSX, breaking the build:
{noformat}bin-package:
[mkdir] Created dir:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1
[mkdir] Created dir:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/bin
[mkdir] Created dir:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/etc/hadoop
[mkdir] Created dir:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/lib
[mkdir] Created dir:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/libexec
[mkdir] Created dir:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/sbin
[mkdir] Created dir:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/share/hadoop/contrib
[mkdir] Created dir:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/share/hadoop/webapps
[mkdir] Created dir:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/share/hadoop/templates/conf
[copy] Copying 11 files to
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/share/hadoop/templates/conf
[copy] Copying 39 files to
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/share/hadoop/lib
[copy] Copying 15 files to
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/share/hadoop/lib
BUILD FAILED
/Users/jhoman/repos/hadoop-common/build.xml:1611:
/Users/jhoman/repos/hadoop-common/build/hadoop-0.20.205.1/native does not exist.
{noformat}
Once one fixes this, one discovers the build is also trying to build the linux
task controller, regardless of whether not the native flag is set, which also
fails.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Should we release 0.20.204.0rc2?
>> This vote is still running with no votes other than mine. The vote was started 9 days ago and, if it kept running after Allen's vote, it would have ended three days ago with the result of 1-0 to release. As such, let's call this release 2Owen3.
[jira] [Resolved] (HADOOP-7458) Namenode not get started! FSNamesystem initialization failed. java.io.FileNotFoundException
[ https://issues.apache.org/jira/browse/HADOOP-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-7458. - Resolution: Invalid This appears to be an issue of file corruption. The best forum for resolving it is the mailing list, rather than the issue tracker. Please write to the hdfs users' list. Closing. > Namenode not get started! FSNamesystem initialization failed. > java.io.FileNotFoundException > --- > > Key: HADOOP-7458 > URL: https://issues.apache.org/jira/browse/HADOOP-7458 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 0.20.2 > Environment: CentOS release 5.5 (Final), 18 node Cluster >Reporter: Sakthivel Murugasamy >Priority: Blocker > Labels: hadoop > Original Estimate: 48h > Remaining Estimate: 48h > > 2011-07-13 12:04:12,967 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed. > java.io.FileNotFoundException: File does not exist: > /opt/data/tmp/mapred/system/job_201107041958_0120/j^@^@^@^@^@^@ > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetPermission(FSDirectory.java:544) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:724) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:992) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:812) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:292) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965) > 2011-07-13 12:04:13,006 ERROR > org.apache.hadoop.hdfs.server.namenode.NameNode: > java.io.FileNotFoundException: File does not exist: > /opt/data/tmp/mapred/system/job_201107041958_0120/j^@^@^@^@^@^@ > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetPermission(FSDirectory.java:544) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:724) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:992) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:812) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:292) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965) > In the path /opt/data/tmp/mapred, "system/" folder itself is not available -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-7457) Remove out-of-date Chinese language documentation
Remove out-of-date Chinese language documentation - Key: HADOOP-7457 URL: https://issues.apache.org/jira/browse/HADOOP-7457 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Jakob Homan The Chinese language documentation haven't been updated (other than copyright years and svn moves) since their original contribution several years ago. Worse than no docs is out-of-date, wrong docs. We should delete them from the source tree. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release candidate 0.20.203.0-rc0
Tested the RC on a single node cluster, kicked the tires. Looks good. +1 on its release. Regardless of how the RC got here, we only get benefit from releasing it. It represents a huge chunk of work from our contributors, provides needed features for our users and moves us one step closer to making regular releases again. -Jakob On Tue, May 3, 2011 at 11:33 AM, Konstantin Boudnik wrote: > Yup, exactly right - it has been reverted in the trunk as well. Thanks > for digging this up, Koji! > > On Tue, May 3, 2011 at 11:22, Koji Noguchi wrote: except HADOOP-6386 and HADOOP-6428. >>> causes a rolling port side effect in TT >>> >> I remember bugging Cos and Rob to revert HADOOP-6386. >> https://issues.apache.org/jira/browse/HADOOP-6760?focusedCommentId=12867342&; >> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#commen >> t-12867342 >> >> Koji >> >> On 5/2/11 9:43 PM, "Konstantin Boudnik" wrote: >> >>> On Mon, May 2, 2011 at 16:56, Arun C Murthy wrote: >>> On May 2, 2011, at 3:01 PM, Arun C Murthy wrote: > On May 2, 2011, at 12:31 PM, Tom White wrote: > >> I just did a quick search, and these are the JIRAs that are in 0.20.2 >> but appear not to be in 0.20.203.0. >> > > Thanks Tom. > > I did a quick analysis: > > # Remaining for 0.20.203 > * HADOOP-5611 > * HADOOP-5612 > * HADOOP-5623 > * HDFS-596 > * HDFS-723 > * HDFS-732 > * HDFS-579 > * MAPREDUCE-1070 > * HADOOP-6315 > * MAPREDUCE-1163 > * HADOOP-5759 > * HADOOP-6269 > * HADOOP-6386 > * HADOOP-6428 > Owen, Suresh and I have committed everything on this list except HADOOP-6386 and HADOOP-6428. Not sure which of the two are relevant/necessary, I'll check with Cos. Other than that hadoop-0.20.203 now a superset of hadoop-0.20.2. >>> >>> I have looked somewhat more into these two JIRAs and if I remember correctly >>> this fix causes a rolling port side effect in TT and it has been reverted in >>> 0.20.200 (Y! Fred? release) because Ops weren't happy about this (I am sure >>> you can check internal Git to cross-verify my recollection). >>> >>> Considering above, these might be better left outside of the release and, >>> perhaps, they should be reverted in trunk as well. >>> >>> Cos >>> >>> thanks, Arun >> >> >
Re: VOTE: Committing HADOOP-6949 to 0.22 branch
+1 n.b. that the vote lost hdfs and common dev at some point. I've added them back. On Tue, Mar 29, 2011 at 9:18 AM, Amit Sangroya wrote: > +1 > > On Tue, Mar 29, 2011 at 6:04 PM, Stephen Boesch wrote: >> +1 >> >> 2011/3/29 Doug Cutting >> >>> +1 >>> >>> I don't think this creates an incompatibility. It changes the RPC wire >>> format, but we already require that clients and servers run identical >>> builds. No application that ran with a prior version of Hadoop would be >>> broken by this change when it upgrades to this version of Hadoop. >>> >>> Doug >>> >>> On 03/28/2011 09:39 PM, Konstantin Shvachko wrote: >>> > HADOOP-6949 introduced a very important optimization to the RPC layer. >>> Based >>> > on the benchmarks presented in HDFS-1583 this provides an order of >>> magnitude >>> > improvement of current RPC implementation. >>> > RPC is a common component of Hadoop projects. Many of them should benefit >>> > from this change. But since this is an incompatible change it requires a >>> > vote to be included into a previous branch. >>> > Please vote for inclusion of this change into branch 0.22. >>> > >>> > +1 from me. >>> > >>> > Thanks, >>> > --Konstantin >>> > >>> >> >
[jira] Resolved: (HADOOP-6236) clean-up Trash.moveToTrash() code. Should throw exception instead of returning false
[ https://issues.apache.org/jira/browse/HADOOP-6236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-6236. - Resolution: Duplicate Resolving as duplicate of HADOOP-6345 > clean-up Trash.moveToTrash() code. Should throw exception instead of > returning false > > > Key: HADOOP-6236 > URL: https://issues.apache.org/jira/browse/HADOOP-6236 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Boris Shkolnik > > In case of errors it should throw an exception instead of returning false. > One valid case is when interval is 0 (trash disabled), but that could be > moved to a separate method isEnabled(). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Reopened: (HADOOP-6463) Uninformative error message for some sort of file permission problem from the hadoop command
[ https://issues.apache.org/jira/browse/HADOOP-6463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reopened HADOOP-6463: - > Uninformative error message for some sort of file permission problem from the > hadoop command > > > Key: HADOOP-6463 > URL: https://issues.apache.org/jira/browse/HADOOP-6463 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 0.20.1 >Reporter: Benson Margulies > Attachments: HADOOP-6463.patch > > > It would be better if this error message would include the pathname for which > permission was denied. > benson@cn0:/u1/benson/hadoop-test$ /u1/hadoop/bin/hadoop jar > /u1/hadoop/hadoop-0.20.1-examples.jar wordcount /users/benson/gutenberg > /users/benson/gutenberg-output > Exception in thread "main" java.io.IOException: Permission denied > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.checkAndCreate(File.java:1716) > at java.io.File.createTempFile(File.java:1804) > at org.apache.hadoop.util.RunJar.main(RunJar.java:115) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-6463) Uninformative error message for some sort of file permission problem from the hadoop command
[ https://issues.apache.org/jira/browse/HADOOP-6463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-6463. - Resolution: Won't Fix > Uninformative error message for some sort of file permission problem from the > hadoop command > > > Key: HADOOP-6463 > URL: https://issues.apache.org/jira/browse/HADOOP-6463 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 0.20.1 >Reporter: Benson Margulies > Attachments: HADOOP-6463.patch > > > It would be better if this error message would include the pathname for which > permission was denied. > benson@cn0:/u1/benson/hadoop-test$ /u1/hadoop/bin/hadoop jar > /u1/hadoop/hadoop-0.20.1-examples.jar wordcount /users/benson/gutenberg > /users/benson/gutenberg-output > Exception in thread "main" java.io.IOException: Permission denied > at java.io.UnixFileSystem.createFileExclusively(Native Method) > at java.io.File.checkAndCreate(File.java:1716) > at java.io.File.createTempFile(File.java:1804) > at org.apache.hadoop.util.RunJar.main(RunJar.java:115) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Add Guava as a dependency?
+1 On Thu, Feb 10, 2011 at 10:04 PM, Konstantin Boudnik wrote: > Actually it seems that Pig uses this already, which perhaps means it > is good enough for us as well ;) > -- > Take care, > Konstantin (Cos) Boudnik > > On Thu, Feb 10, 2011 at 19:45, Todd Lipcon wrote: >> Anyone mind if I pull in the Guava library as a dependency for Common? It >> has a bunch of very useful utilities - in this particular case the one I'm >> itching to use is ThreadFactoryBuilder: >> >> http://guava-libraries.googlecode.com/svn/tags/release05/javadoc/com/google/common/util/concurrent/ThreadFactoryBuilder.html >> >> More info here: >> http://code.google.com/p/guava-libraries/ >> >> -Todd >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> >
Re: Do Common developers use FindBugs?
Cesar- Theoretically, Findbugs is run as part of the test-patch process, a correctness verification that all patches need to go through before being committed, for all the projects. However, test-patch has a checkered history and has not always been reliable. So yes, we do, when we do. -jg 2011/1/24 César Couto : > Dear developers, > > I am a PhD student at UFMG, Brazil and as part of my research I am > making a study about the relevance of the warnings reported by the > FindBugs bug finding tool. > > Since I am planning to use Common as a subject system in my research, > I would like to know if Common's developers usually run FindBugs as > > part of the system development process. > > Thanks in advance, > > Cesar Couto > > -- > http://www.decom.cefetmg.br/cesar >
Re: Ready for review?
Niels- Thanks for the contribution. For the moment your task is done. Now it's up to a committer to review the patch and, either provide you with feedback for its improvement, or commit it. It's in the patch available state, which is the flag for reviewers to know there's work for them to do. Since this is a volunteer effort, I'm afraid there's no firm timeline for when this will get done. -Jakob On Fri, Jan 7, 2011 at 6:50 AM, Niels Basjes wrote: > Hi, > > I consider the patch I created to be ready for review by a code reviewer. > > It does what I want it to do and Hudson gives an overall +1. > The http://wiki.apache.org/hadoop/HowToContribute was unclear to me on > what to do next. > > So, what should I do? > Simply wait / change the stated of the issue / ... > > https://issues.apache.org/jira/browse/HADOOP-7076 > > -- > Met vriendelijke groeten, > > Niels Basjes >
Re: Jira workflow problem.
Niels- For new patches, just cancel the current and submit will become available again. Also, there's no need to delete prior versions of the patch; it's better to keep them around for posterity and comparison. Hudson will run against whatever is uploaded most recently (this is important for backports, etc.) -Jakob On Thu, Jan 6, 2011 at 1:41 PM, Niels Basjes wrote: > I seem to have selected the wrong option in Jira to get the latest > patch handled. > For some reason the option to indicate a new patch has been made > available is no longer present. > > https://issues.apache.org/jira/browse/HADOOP-7076 > > What did I do wrong and what can I do to fix this? > > Thanks. > -- > Met vriendelijke groeten, > > Niels Basjes >
[jira] Created: (HADOOP-7069) Replace forrest with supported framework
Replace forrest with supported framework Key: HADOOP-7069 URL: https://issues.apache.org/jira/browse/HADOOP-7069 Project: Hadoop Common Issue Type: Improvement Components: documentation Reporter: Jakob Homan Fix For: 0.23.0 It's time to burn down the forrest. Apache forrest, which is used to generate the documentation for all three subprojects, has not had a release in several years (0.8, the version we use was released April 18, 2007), and requires JDK5, which was EOL'ed in November 2009. Since it doesn't seem likely Forrest will be developed any more, and JDK5 is not shipped with recent OSX versions, or included by default in most linux distros, we should look to find a new documentation system and convert the current docs to it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6989) TestSetFile is failing on trunk
TestSetFile is failing on trunk --- Key: HADOOP-6989 URL: https://issues.apache.org/jira/browse/HADOOP-6989 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.22.0 Reporter: Jakob Homan Fix For: 0.22.0 Testsuite: org.apache.hadoop.io.TestSetFile Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 1.015 sec - Standard Output --- 2010-10-04 16:32:01,030 INFO io.TestSetFile (TestSetFile.java:generate(56)) - generating 1 records in memory 2010-10-04 16:32:01,249 INFO io.TestSetFile (TestSetFile.java:generate(63)) - sorting 1 records 2010-10-04 16:32:01,350 INFO io.TestSetFile (TestSetFile.java:writeTest(72)) - creating with 1 records - --- Testcase: testSetFile took 0.964 sec Caused an ERROR key class or comparator option must be set java.lang.IllegalArgumentException: key class or comparator option must be set at org.apache.hadoop.io.MapFile$Writer.(MapFile.java:247) at org.apache.hadoop.io.SetFile$Writer.(SetFile.java:60) at org.apache.hadoop.io.TestSetFile.writeTest(TestSetFile.java:73) at org.apache.hadoop.io.TestSetFile.testSetFile(TestSetFile.java:45) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6987) Use JUnit Rule to optionally fail test cases that run more than 10 seconds
Use JUnit Rule to optionally fail test cases that run more than 10 seconds -- Key: HADOOP-6987 URL: https://issues.apache.org/jira/browse/HADOOP-6987 Project: Hadoop Common Issue Type: Improvement Components: test Affects Versions: 0.21.0 Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.22.0 Using JUnit Rules annotations we can fail tests cases that take longer than 10 seconds (for instance) to run. This provides a regression check against test cases taking longer than they had previously due to unintended code changes, as well as provides a membership criteria for unit tests versus integration tests in HDFS and MR. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6892) Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters)
Common component of HDFS-1150 (Verify datanodes' identities to clients in secure clusters) -- Key: HADOOP-6892 URL: https://issues.apache.org/jira/browse/HADOOP-6892 Project: Hadoop Common Issue Type: New Feature Components: security Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.22.0 HDFS-1150 will have changes to the start-up scripts and HttpServer. These are handled here. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6853) Common component of HDFS-1045
Common component of HDFS-1045 - Key: HADOOP-6853 URL: https://issues.apache.org/jira/browse/HADOOP-6853 Project: Hadoop Common Issue Type: Bug Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.22.0 HDFS-1045 modified UGI, which is in Common on trunk. This JIRA is for that change. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6824) Improve NetUtils:normalizeHostName
Improve NetUtils:normalizeHostName -- Key: HADOOP-6824 URL: https://issues.apache.org/jira/browse/HADOOP-6824 Project: Hadoop Common Issue Type: Improvement Reporter: Jakob Homan HADOOP-6682 revealed a bug in NetUtils:normalizeHostName, which got fixed, but it may be worth replacing the current best-effort code with a more rigorous approach, as Hong suggested. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6823) Provide validation of security-related settings on startup
Provide validation of security-related settings on startup -- Key: HADOOP-6823 URL: https://issues.apache.org/jira/browse/HADOOP-6823 Project: Hadoop Common Issue Type: Improvement Affects Versions: 0.22.0 Reporter: Jakob Homan Priority: Minor There are several new config settings for security, involving keytab files, kerberos principals, security filters, etc. that need to be set for a secure cluster to come up. It'd be nice if on startup, we checked that the required config values have been set and have something vaguely reasonable for values. Displaying this information as well would provide a good summary for those trying to debug cluster security issues. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6822) Provide information as to whether or not security is enabled on web interface
Provide information as to whether or not security is enabled on web interface - Key: HADOOP-6822 URL: https://issues.apache.org/jira/browse/HADOOP-6822 Project: Hadoop Common Issue Type: Sub-task Reporter: Jakob Homan -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6661) User document for UserGroupInformation.doAs
[ https://issues.apache.org/jira/browse/HADOOP-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-6661. - Resolution: Fixed I've committed this. Thanks, Jitendra! Resolving as fixed. > User document for UserGroupInformation.doAs > --- > > Key: HADOOP-6661 > URL: https://issues.apache.org/jira/browse/HADOOP-6661 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: HADOOP-6661-y20.1.patch, HADOOP-6661-y20.2.patch, > HADOOP-6661.1.patch, HADOOP-6661.2.patch, HADOOP-6661.3.patch, > HADOOP-6661.4.patch > > > A user document should be added for secure impersonation feature. The > document should cover a code example about how UserGroupInformation.doAs > should be used for secure impersonation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6741) Refactor Hadoop Command line interface.
[ https://issues.apache.org/jira/browse/HADOOP-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-6741. - Resolution: Duplicate I'll go ahead and say this duplicates HADOOP-6425, but that being said, HADOOP-6425 could probably be split into several subtasks that may be more manageable. > Refactor Hadoop Command line interface. > --- > > Key: HADOOP-6741 > URL: https://issues.apache.org/jira/browse/HADOOP-6741 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Ravi Phulari > Fix For: 0.22.0 > > > Currently Hadoop fs commands are scattered between FsShell , FsPermissions > and DfsAdmin . It will be great if refactor hadoop CLI for better debugging, > performance and code maintenance. > Any thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6725) Evaluate HtmlUnit for unit and regression testing webpages
Evaluate HtmlUnit for unit and regression testing webpages -- Key: HADOOP-6725 URL: https://issues.apache.org/jira/browse/HADOOP-6725 Project: Hadoop Common Issue Type: Test Components: test Reporter: Jakob Homan Priority: Minor HtmlUnit (http://htmlunit.sourceforge.net/) looks like it may be a good tool to help unit testing and evaluating our various webpages throughout the project. Currently this is done only occasionally in the code (usually falls to being a manual test during release cycles), and when it is done, usually the code to parse the webpage, etc. is re-written each time. The framework is Apache licensed, so including it won't be an issue. If it's found to be useful, new JIRAs for HDFS and MR should be opened. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6682) NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly
NetUtils:normalizeHostName does not process hostnames starting with [a-f]
correctly
---
Key: HADOOP-6682
URL: https://issues.apache.org/jira/browse/HADOOP-6682
Project: Hadoop Common
Issue Type: Bug
Components: io
Reporter: Jakob Homan
public static String normalizeHostName(String name) {
if (Character.digit(name.charAt(0), 16) != -1) {
return name;
This code is attempting to short-circuit the hostname->ip resolution on the
assumption that if name starts with a digit, it's already an ip address. This
is of questionable value, but because it checks for a hex digit, it will fail
on names starting with [a-f]. Such names will not be converted to an ip
address, but be returned unchanged.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [VOTE] HADOOP-6671 - To use maven for hadoop common build
+1 for the the exploration. *If* it works as advertised, it'll be a great time saver. Giridharan Kesavan wrote: I would like call for a vote to created a development branch of common trunk to work on mavenizing hadoop common. -Giri
[jira] Resolved: (HADOOP-6611) Fix javac warnings introduced by HADOOP-6584
[
https://issues.apache.org/jira/browse/HADOOP-6611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan resolved HADOOP-6611.
-
Resolution: Invalid
This patch should be attached to 6584, since the trunk patch won't have these
issues...
> Fix javac warnings introduced by HADOOP-6584
>
>
> Key: HADOOP-6611
> URL: https://issues.apache.org/jira/browse/HADOOP-6611
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
> Reporter: Jakob Homan
>Assignee: Jakob Homan
>
> Some javac warnings slipped through in HADOOP-6584:
> {noformat}[exec] [javadoc]
> /Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/http/HttpServer.java:45
> 4: warning - @param argument "needClientAuth" is not a parameter name.
> [exec] [javadoc]
> /Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/security/Krb5AndCertsSs
> lSocketConnector.java:183: warning - Tag @link: reference not found:
> Krb5SslSocketConnector
> [exec] [javadoc]
> /Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/security/Krb5AndCertsSs
> lSocketConnector.java:183: warning - Tag @link: reference not found:
> Krb5SslSocketConnector
> [exec] [javadoc]
> /Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/security/Krb5AndCertsSs
> lSocketConnector.java:183: warning - Tag @link: reference not found:
> Krb5SslSocketConnector
> [exec] [javadoc] Building index for all the packages and classes...
> [exec] [javadoc]
> /Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/security/Krb5AndCertsSs
> lSocketConnector.java:183: warning - Tag @link: reference not found:
> Krb5SslSocketConnector
> [exec] [javadoc] Building index for all classes...
> [exec] [javadoc] Generating
> /Users/jhoman/work/git/hadoop-common/build/docs/api/stylesheet.css...
> [exec] [javadoc] 5 warnings{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6611) Fix javac warnings introduced by HADOOP-6584
Fix javac warnings introduced by HADOOP-6584
Key: HADOOP-6611
URL: https://issues.apache.org/jira/browse/HADOOP-6611
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Jakob Homan
Assignee: Jakob Homan
Some javac warnings slipped through in HADOOP-6584:
{noformat}[exec] [javadoc]
/Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/http/HttpServer.java:45
4: warning - @param argument "needClientAuth" is not a parameter name.
[exec] [javadoc]
/Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/security/Krb5AndCertsSs
lSocketConnector.java:183: warning - Tag @link: reference not found:
Krb5SslSocketConnector
[exec] [javadoc]
/Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/security/Krb5AndCertsSs
lSocketConnector.java:183: warning - Tag @link: reference not found:
Krb5SslSocketConnector
[exec] [javadoc]
/Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/security/Krb5AndCertsSs
lSocketConnector.java:183: warning - Tag @link: reference not found:
Krb5SslSocketConnector
[exec] [javadoc] Building index for all the packages and classes...
[exec] [javadoc]
/Users/jhoman/work/git/hadoop-common/src/core/org/apache/hadoop/security/Krb5AndCertsSs
lSocketConnector.java:183: warning - Tag @link: reference not found:
Krb5SslSocketConnector
[exec] [javadoc] Building index for all classes...
[exec] [javadoc] Generating
/Users/jhoman/work/git/hadoop-common/build/docs/api/stylesheet.css...
[exec] [javadoc] 5 warnings{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6603) Provide workaround for issue with Kerberos not resolving cross-realm principal
Provide workaround for issue with Kerberos not resolving cross-realm principal -- Key: HADOOP-6603 URL: https://issues.apache.org/jira/browse/HADOOP-6603 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Jakob Homan Java's SSL-Kerberos implementation does not correctly obtain the principal for cross-realm principles when clients initiate connections to servers, resulting in the client being unable to authenticate the server. We need a work-around until this bug gets fixed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6594) Update hdfs script to provide fetchdt tool
Update hdfs script to provide fetchdt tool -- Key: HADOOP-6594 URL: https://issues.apache.org/jira/browse/HADOOP-6594 Project: Hadoop Common Issue Type: New Feature Reporter: Jakob Homan Assignee: Jakob Homan Since the bin/hdfs script is still in common, this JIRA is needed to provide access to the new delegation token fetcher done in HDFS-994 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6584) Provide Kerberized SSL encryption for webservices
Provide Kerberized SSL encryption for webservices
-
Key: HADOOP-6584
URL: https://issues.apache.org/jira/browse/HADOOP-6584
Project: Hadoop Common
Issue Type: Improvement
Reporter: Jakob Homan
Assignee: Jakob Homan
Some web services should be authenticated via SSL backed by Kerberos, both to
provide cross-cluster secure communication and to provide intra-cluster server
authentication for services such as the
{Name,SecondaryName,Backup,Checkpoint}node's image transfer and balancer.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6527) UserGroupInformation::createUserForTesting clobbers already defined group mappings
UserGroupInformation::createUserForTesting clobbers already defined group
mappings
--
Key: HADOOP-6527
URL: https://issues.apache.org/jira/browse/HADOOP-6527
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Jakob Homan
In UserGroupInformation::createUserForTesting the follow code creates a new
groups instance, obliterating any groups that have been previously defined in
the static groups field.
{code}if (!(groups instanceof TestingGroups)) {
groups = new TestingGroups();
}
{code}
This becomes a problem in tests that start a Mini{DFS,MR}Cluster and then
create a testing user. The user that started the user (generally the real user
running the test) immediately has their groups wiped out and is prevented from
accessing files/folders/queues they should be able to. Before the
UserGroupInformation.createRemoteUserForTesting, calls to userA.getGroups may
return {"a", "b", "c"} and immediately after the new fake user is created, the
same call will return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6521) FsPermission:SetUMask not updated to use new-style umask setting.
FsPermission:SetUMask not updated to use new-style umask setting.
-
Key: HADOOP-6521
URL: https://issues.apache.org/jira/browse/HADOOP-6521
Project: Hadoop Common
Issue Type: Bug
Reporter: Jakob Homan
FsPermission:
221 /** Set the user file creation mask (umask) */
222 public static void setUMask(Configuration conf, FsPermission umask) {
223 conf.setInt(UMASK_LABEL, umask.toShort());
224 }
Needs to be updated to not use a decimal value. This is a bug introduced by
HADOOP-6234.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6511) FsPermission.getDefault choice of 0777 is bad in all cases
[
https://issues.apache.org/jira/browse/HADOOP-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan resolved HADOOP-6511.
-
Resolution: Invalid
Hearing no counter-arguments, I'm closing this as invalid...
> FsPermission.getDefault choice of 0777 is bad in all cases
> --
>
> Key: HADOOP-6511
> URL: https://issues.apache.org/jira/browse/HADOOP-6511
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 0.20.1
> Environment: hadoop 20.1, java 1.6.0_17, Fedora
>Reporter: robert Cook
>
> /** Get the default permission. */
> 227 public static FsPermission getDefault() {
> 228 return new FsPermission((short)00777);
> 229 }
> setting any data file to execute-others is bad security under all
> circumstances
> should be 664 or 644
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5436) job history directory grows without bound, locks up job tracker on new job submission
[ https://issues.apache.org/jira/browse/HADOOP-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-5436. - Resolution: Won't Fix > job history directory grows without bound, locks up job tracker on new job > submission > - > > Key: HADOOP-5436 > URL: https://issues.apache.org/jira/browse/HADOOP-5436 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.19.0 >Reporter: Tim Williamson > Attachments: HADOOP-5436.patch > > > An unpleasant surprise upgrading to 0.19: requests to jobtracker.jsp would > take a long time or even time out whenever new jobs where submitted. > Investigation showed the call to JobInProgress.initTasks() was calling > JobHistory.JobInfo.logSubmitted() which in turn was calling > JobHistory.getJobHistoryFileName() which was pegging the CPU for a couple > minutes. Further investigation showed the were 200,000+ files in the job > history folder -- and every submission was creating a FileStatus for them > all, then applying a regular expression to just the name. All this just on > the off chance the job tracker had been restarted (see HADOOP-3245). To make > matters worse, these files cannot be safely deleted while the job tracker is > running, as the disappearance of a history file at the wrong time causes a > FileNotFoundException. > So to summarize the issues: > - having Hadoop default to storing all the history files in a single > directory is a Bad Idea > - doing expensive processing of every history file on every job submission is > a Worse Idea > - doing expensive processing of every history file on every job submission > while holding a lock on the JobInProgress object and thereby blocking the > jobtracker.jsp from rendering is a Terrible Idea (note: haven't confirmed > this, but a cursory glance suggests that's what's going on) > - not being able to clean up the mess without taking down the job tracker is > just Unfortunate -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-5436) job history directory grows without bound, locks up job tracker on new job submission
[ https://issues.apache.org/jira/browse/HADOOP-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reopened HADOOP-5436: - > job history directory grows without bound, locks up job tracker on new job > submission > - > > Key: HADOOP-5436 > URL: https://issues.apache.org/jira/browse/HADOOP-5436 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.19.0 >Reporter: Tim Williamson > Attachments: HADOOP-5436.patch > > > An unpleasant surprise upgrading to 0.19: requests to jobtracker.jsp would > take a long time or even time out whenever new jobs where submitted. > Investigation showed the call to JobInProgress.initTasks() was calling > JobHistory.JobInfo.logSubmitted() which in turn was calling > JobHistory.getJobHistoryFileName() which was pegging the CPU for a couple > minutes. Further investigation showed the were 200,000+ files in the job > history folder -- and every submission was creating a FileStatus for them > all, then applying a regular expression to just the name. All this just on > the off chance the job tracker had been restarted (see HADOOP-3245). To make > matters worse, these files cannot be safely deleted while the job tracker is > running, as the disappearance of a history file at the wrong time causes a > FileNotFoundException. > So to summarize the issues: > - having Hadoop default to storing all the history files in a single > directory is a Bad Idea > - doing expensive processing of every history file on every job submission is > a Worse Idea > - doing expensive processing of every history file on every job submission > while holding a lock on the JobInProgress object and thereby blocking the > jobtracker.jsp from rendering is a Terrible Idea (note: haven't confirmed > this, but a cursory glance suggests that's what's going on) > - not being able to clean up the mess without taking down the job tracker is > just Unfortunate -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-6500) Only the S3 test extends FileSystemContractBaseTest; the other FS tests do not
[ https://issues.apache.org/jira/browse/HADOOP-6500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-6500. - Resolution: Duplicate This issue is currently being handled by HDFS-303. Closing as duplicate. > Only the S3 test extends FileSystemContractBaseTest; the other FS tests do not > -- > > Key: HADOOP-6500 > URL: https://issues.apache.org/jira/browse/HADOOP-6500 > Project: Hadoop Common > Issue Type: Test > Components: test >Affects Versions: 0.20.1 > Environment: Fedora, java 1.6.0_17, hadoop 20.1 >Reporter: robert Cook > > TestChecksumFileSystem etc does not extend FileSystemContractBaseTest > In fact, ChecksumFileSystem doesn't pass the ContractBaseTest -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-6385) dfs does not support -rmdir (was HDFS-639)
[ https://issues.apache.org/jira/browse/HADOOP-6385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reopened HADOOP-6385: - Hey Scott, re-opening as resolved is used for issues that have been committed, which this hasn't. Were you still wanting this feature? I personally have some concerns about it, but haven't had a chance to look at the patch or comment. > dfs does not support -rmdir (was HDFS-639) > -- > > Key: HADOOP-6385 > URL: https://issues.apache.org/jira/browse/HADOOP-6385 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Reporter: Scott Phillips >Priority: Minor > Attachments: HADOOP-6385.patch > > > From HDFS-639 > > Given we have a mkdir, we should have a rmdir. Using rmr is not > > a reasonable substitute when you only want to delete empty > > directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6396) Provide a description in the exception when an error is encountered parsing umask
Provide a description in the exception when an error is encountered parsing umask - Key: HADOOP-6396 URL: https://issues.apache.org/jira/browse/HADOOP-6396 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.21.0, 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.21.0, 0.22.0 Currently when there is a problem parsing a umask, the exception text is just the offending umask with no other clue, which can be quite confusing as demonstrated in HDFS-763. This message should include the nature of the problem and whether or not the umask parsing attempt was using old-style or new-style values. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: HDFS-758 in Hadoop-21 , Updates to Namenode health page
+1. Backporting this does not in any way impact the release of 21. -Jakob Hairong Kuang wrote: +1. Although this is a new feature, I'd like to have it committed to 0.21 since we have so many issues with delayed decomission recently. Hairong On 11/24/09 6:06 PM, "Suresh Srinivas" wrote: +1. This will also help debug the issues when decommissioning takes a long time to complete. On 11/23/09 7:36 PM, "Jitendra Nath Pandey" wrote: Hi, We will be committing some changes to the Namenode Health page (dfshealth.jsp) as part of the fix in HDFS-758. This will enable us to monitor the progress of decommissioning of datanodes more effectively. Summary of changes : 1. A new link on the page for Decommissioning nodes. 2. This link will point to a new page with details about decommissioning status for each node which include a) Number of under-relplicated blocks in the node. b) Number of blocks with only no live replica (i.e. All its replicas are on decommissioning nodes). c) Number of under-replicated blocks in open files. d) Time since decommissioning started. 3. The main page will also contain total number of under-replicated blocks in the cluster. Thanks jitendra
[jira] Resolved: (HADOOP-6355) Make FileSystemContractBaseTest, LocalFileSystem, and FileSystem javadoc compliant
[ https://issues.apache.org/jira/browse/HADOOP-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-6355. - Resolution: Duplicate I'm going to go ahead and say this does duplicate the other jira. It would certainly be great to iron out those differences, but we ran into problems involving backwards compatibility when we worked on it previously. If possible, stick to the FileSystemContract test and then rely on DFS behavior as guidance if there are differences. > Make FileSystemContractBaseTest, LocalFileSystem, and FileSystem javadoc > compliant > --- > > Key: HADOOP-6355 > URL: https://issues.apache.org/jira/browse/HADOOP-6355 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Reporter: Gregory Farnum >Priority: Minor > > These do not all seem to play nicely together. ie, plugging the > LocalFileSystem into FileSystemContractBaseTest (via a subclass that just > overrides setUp to set fs=LocalFileSystem) results in 5 failures and 19 > errors out of 28 tests. > Additionally, the unit tests expect a few undocumented behaviors (ie, it > expects to get an IOException on mkdirs(subdir/of/a/file) rather than just > returning false) and some contradictory behaviors (it expects that > delete(directory, false) should succeed on an empty directory) compared to > FileSystem's javadoc. > I suspect that most of the fixes should come from changing the unit tests to > be more sensible rather than modifying [Local]FileSystem, but however it's > done these should all agree on expected behavior! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6345) Refactor Trash::moveToTrash() and its accompanying poor tests
Refactor Trash::moveToTrash() and its accompanying poor tests - Key: HADOOP-6345 URL: https://issues.apache.org/jira/browse/HADOOP-6345 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Jakob Homan We've had several issues relating to the Trash and its access via the shell, none of which were picked up by the unit tests. The current moveToTrash method has 8 different ways it can terminate and sometimes uses a false value and sometimes uses an exception to indicate failure. This method should be refactored to improve readability and testability, and new tests written to exercise all possible code paths. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-3474) Need an option -rm and -rmr to *not* move to Trash, but actually delete
[ https://issues.apache.org/jira/browse/HADOOP-3474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HADOOP-3474. - Resolution: Duplicate This ended up being duplicated by HADOOP-6080. Resolving. > Need an option -rm and -rmr to *not* move to Trash, but actually delete > --- > > Key: HADOOP-3474 > URL: https://issues.apache.org/jira/browse/HADOOP-3474 > Project: Hadoop Common > Issue Type: New Feature > Components: fs >Affects Versions: 0.17.0 >Reporter: Milind Bhandarkar > > By default, when Trash is enabled, -rm and -rmr just move files/directories > to .Trash, which are actually deleted after some time set in the config. > When DFS gets full, we need to quickly create space on it by deleting files, > so it will be good to have a --force option to both -rm and -rmr, so that > files are actually deleted and not moved. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6337) Update FilterInitializer class to be more visible and take a conf for further development
Update FilterInitializer class to be more visible and take a conf for further development - Key: HADOOP-6337 URL: https://issues.apache.org/jira/browse/HADOOP-6337 Project: Hadoop Common Issue Type: New Feature Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.22.0 Currently the FilterInitializer class, used to filter access to the Hadoop web interfaces, has its main method, initFilter, set to package private. This means that any classes wishing to implement this type must be in the same package. This should be public so FilterInitializers can reside in other packages. Also, currently all parameters to the FilterInitializer must be provided at compile time (or via a different configuration method). It would be better if the FilterInitalizer::initFilter received a Configuration parameter so it can pull conf values out at run-time. Alternatively, the class could implement Configured, but that seems heavier than is needed at this point. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Developing Hadoop and HDFS
Thanks for your interest, Geoff. Yes, finding open JIRAS and contributing patches is very helpful. We also maintain a wishlist of projects that one could work on: http://wiki.apache.org/hadoop/ProjectSuggestions. In addition, please do consider documentation and example work as well, as this is very helpful both to new users and developers starting on the project. Thanks, Jakob Hadoop at Yahoo! Geoffrey Gallaway wrote: Hello, Yes, another person looking to contribute to and develop Hadoop. I'm looking to start off small, fixing a few bugs before moving into larger stuff. First, a bit of background: Years ago I had the idea of creating a semi-decentralized distributed file system. The idea came when I was working for a small/medium sized company who was looking for a simple backup solution for their workstations. PC's back then came with 100+ GB hard drives but, as simple workstations, employees were using less than half that space. Why not have each workstation backup to a few other workstations, duplicating files across multiple machines for redundancy. RAID for the network. I started coming up with design and architecture specs, protocol examples and even started writing a bit of the system (in Java). I tried to find a few interested developers but everyone seemed to think the task was much too large to be accomplished as a side project (and I didn't think, given the IT industry of the time, that anyone would fund it). Later, I realized such a distributed system could be much more than a simple file backup solution. It looks like Hadoop and HDFS are creating a lot of what I had wanted to create, it's already surpassed what I had in mind in most ways. So, where should I start? Just start fixing bugs listed in JIRA? Geoff
[jira] Created: (HADOOP-6252) Provide method to determine if a deprecated key was set in the config file
Provide method to determine if a deprecated key was set in the config file -- Key: HADOOP-6252 URL: https://issues.apache.org/jira/browse/HADOOP-6252 Project: Hadoop Common Issue Type: Improvement Components: conf Affects Versions: 0.21.0 Reporter: Jakob Homan Assignee: Jakob Homan HADOOP-6105 provided a method to deprecate config keys and transparently refer to the new key. However, it didn't provide a method to see if the deprecated key had been used in the config file. This is useful when, if the deprecated key had been used, its value needs to be converted before use, for instance when we changed the umask format. A method like "boolean wasDeprecatedKeySet()" would be great. Patch shortly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6246) Update umask code to use key deprecation facilities from HADOOP-6105
Update umask code to use key deprecation facilities from HADOOP-6105 Key: HADOOP-6246 URL: https://issues.apache.org/jira/browse/HADOOP-6246 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Jakob Homan Assignee: Jakob Homan When the new octal/symbolic umask JIRA (HADOOP-6234) went through, the config key deprecation patch (HADOOP-6105) wasn't ready. Now that both are committed, the home-brewed key deprecation system from 6234 should be updated to use the new system. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6203) Improve error message when moving to trash fails due to quota issue
Improve error message when moving to trash fails due to quota issue
---
Key: HADOOP-6203
URL: https://issues.apache.org/jira/browse/HADOOP-6203
Project: Hadoop Common
Issue Type: Improvement
Components: fs
Reporter: Jakob Homan
HADOOP-6080 provided an option for deleting files even when overquota, but the
error message that's returned in this situation is unhelpful and doesn't
suggest skipTrash as a remediation:
{noformat}$ hdfs -rmr /foo/bar/bat/boo
rmr: Failed to move to trash:
hdfs://cluster/foo/bar/bat/boo{noformat}
In this situation, the error message should say there was a quote problem and
suggest -skipTrash.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6201) FileSystem::ListStatus should throw FileNotFoundException
FileSystem::ListStatus should throw FileNotFoundException - Key: HADOOP-6201 URL: https://issues.apache.org/jira/browse/HADOOP-6201 Project: Hadoop Common Issue Type: Improvement Components: fs Reporter: Jakob Homan As discussed in HDFS-538, it would be better for FileSystem and its implementations to throw FileNotFoundException when the file is not found, rather than returning null. This will bring it in line with getFileStatus and, I expect, default user expectations. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HADOOP-6164) Test-patch's method of checking for new tests is not correct
Test-patch's method of checking for new tests is not correct Key: HADOOP-6164 URL: https://issues.apache.org/jira/browse/HADOOP-6164 Project: Hadoop Common Issue Type: Bug Components: test Reporter: Jakob Homan As happened in HDFS-458, test-patch/hudson saw the previous patch(HDFS-492) add a new test, bringing the total number of tests to 48. Then it tested 458, which had no new tests as it was a change to the build file. 492 had not yet been committed yet, so there were still only 48 tests in trunk. But test-patch failed the patch, expecting 49 tests. test-patch should check the correct number of tests based on trunk, not on whatever number it last saw. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Developing cross-component patches post-split
Todd Lipcon wrote:
>> If someone could write up some kind of "post-split transition guide"
>> on the Wiki I think that would be generally appreciated.
Here's something I wrote up for re-setting up Eclipse after the split in
a way that gives relatively seamless access to the various projects'
source code. If it's works for other people, it could be part of the wiki.
Setting up Eclipse post-dev:
I believe my Eclipse dev environment is set up for near-seamless
inter-project development following the Great Split of 2009. Here's how
I did it, step-by-step, with no guarantee as to orthodoxy, correctness
or appropriateness. Please let me know, or update, with corrections.
Obligatory works-on-my-machine and YMMV.
With these instructions, you'll be able to work on all three projects at
once. For instance, after the split, trying to look at the source of a
class defined in Common, such as Text, will show you the bytecode in the
included jar file. You can attach the source for review purposes, but
this still leaves the problem of eclipse only running the code from the
jar, rather than from the other project, in the event that you've
modified both. These instructions fix this problem so that any changes,
say in Common, are also applied to HDFS or MapReduce?.
These instructions assume svn and specifically svn from the command
line. They will work fine with git once those repos are set up and also
will work with the svn or git plugin from within eclipse.
1. Import each of the new repositories. Here're the directories that I
picked.
svn checkout https://svn.apache.org/repos/asf/hadoop/common/trunk
hadoop-common
svn checkout https://svn.apache.org/repos/asf/hadoop/hdfs/trunk hadoop-hdfs
svn checkout https://svn.apache.org/repos/asf/hadoop/mapreduce/trunk
hadoop-mapreduce
2. Start Eclipse. For each of the new directories, create a new java
project using that directory name, allowing Eclipse to do its standard
work of importing the project. It was previously necessary to change
Eclipse's default build directory (bin) to something else to keep it
from wiping out Hadoop's bin directory. At the moment, only the common
project has a bin directory. However, I still changed each of my eclipse
build directories (set on the second screen of the new project wizard)
to build/eclipse-files, just in case there is a bin added in the future
and as it's tidier.
3. Ensure that ANT_HOME is defined in Eclipse's environment variables
(Preferences -> Java -> Build Path -> Classpath Variables). Mine is set
to the standard /usr/share/ant
4. From the Navigator window, right click on the build.xml and (#2 Run
as ant build). Among the targets, specify compile,
compile-{common,hdfs,mapred}-test, and eclipse-files. Let Eclipse do its
work and after it's done, each of the projects should compile
successfully and be working correctly independently.
5. To allow the projects to call directly into each other's code, rather
than relying on the bundled libraries, connect each of the projects as
dependencies. For each project set up the natural dependency (hdfs
relies on common, mapred relies on common and hdfs). Right click on the
each project and go build path -> configure build path -> projects tab.
For HDFS, add common. For MapReduce, add Common and HDFS. Unfortunately,
you can't just add everything to everything, as Eclipse detects this as
a cycle and throws up errors.
6. Finally, in order to force Eclipse to look at the other projects'
source code, rather than the included jar files, remove the jar files
from the build path of each project, respectively. From HDFS, remove the
common(core) jar from the build path. From MapReduce, remove the hdfs
and common(core) jars from the build path. HDFS still has a dependency
on MapReduce for tests, and I couldn't get Eclipse to allow me to remove
the MapReduce jar from HDFS project; if anyone figures out how, please
update this document or let me know.
7. All should be well. Now, for instance, if you control-click on a
class defined in Common from hdfs (say Text), you are brought to its
definition in the common project, as expected. Also, if you modify code
in common and run a test from hdfs, it'll pull in the modified code from
common. This doesn't solve the problem of needing to generated patches
for each of the projects if your changes affect each of them, however.
[jira] Commented: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725696#action_12725696
]
Jakob Homan commented on HADOOP-6080:
-
bq. Should we consider excluding trash for files deleted from /tmp (ie make
-skipTrash implicit when deleting from /tmp.)?
I'm not a fan of special cases for certain directories, even for /tmp, and
particularly when we're already straying away from the posix world with the
trash feature. Minimizing surprise seems a good goal, and I'd be very
surprised if I were accustomed to explicitly skipping the trash when I want and
discovering something I had expected to be trashed had been helpfully nuked by
the system.
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
>Assignee: Jakob Homan
> Attachments: HADOOP-6080-v20.patch, HADOOP-6080.patch,
> javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12725659#action_12725659
]
Jakob Homan commented on HADOOP-6080:
-
Updated patches are good to go on all commons unit tests for trunk and all
tests for v20. Test-patch is fine except the incorrect javac warnings, which
are not related.
{noformat} [exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] -1 javac. The applied patch generated 64 javac compiler
warnings (more than the trunk's current 124 warnings).
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.{noformat}
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
>Assignee: Jakob Homan
> Attachments: HADOOP-6080-v20.patch, HADOOP-6080.patch,
> javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-6080:
Attachment: HADOOP-6080.patch
HADOOP-6080-v20.patch
Ran into a problem that I didn't notice with the HDFS version of TestTrash. I
think there's an issue with the FileSystem.listStatus methods between
LocalFileSystem and DistributedFileSystem, which I'll look into. In the
meantime, modified test so that it doesn't rely on that method and works on
both local and distributed file systems.
Will run full test suite tonight, report tomorrow morning. Also, deleted old
patches to avoid confusion. New patches for both trunk and v20 should be good
to go.
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Reporter: Koji Noguchi
>Assignee: Jakob Homan
> Attachments: HADOOP-6080-v20.patch, HADOOP-6080.patch,
> javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-6080:
Attachment: (was: HADOOP-6080-v20.patch)
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
> Assignee: Jakob Homan
> Attachments: javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-6080:
Attachment: (was: HADOOP-6080.patch)
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
> Assignee: Jakob Homan
> Attachments: javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-6080:
Attachment: (was: HADOOP-6080.patch)
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
> Assignee: Jakob Homan
> Attachments: javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-6080:
Status: Open (was: Patch Available)
Canceling patch to double check something.
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
> Assignee: Jakob Homan
> Attachments: HADOOP-6080-v20.patch, HADOOP-6080.patch,
> HADOOP-6080.patch, javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-6080:
Attachment: HADOOP-6080-v20.patch
HADOOP-6080.patch
Attaching two new files:
* Updated patch. Previous patch missed updating the help text for rmr to
include -skipTrash option. No change to actual code.
* Patch for Hadoop 20 off of the Hadoop-20 branch from svn. Nothing had to
be changed for patch, just file locations were different. Code is still the
same. Passes unit tests.
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
> Assignee: Jakob Homan
> Attachments: HADOOP-6080-v20.patch, HADOOP-6080.patch,
> HADOOP-6080.patch, javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-6080:
Attachment: javac_warnings_diff.txt
Test-patch:
{noformat}[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified
tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] -1 javac. The applied patch generated 64 javac compiler warnings
(more than the trunk's current 124 warnings).
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total
number of release audit warnings.
{noformat}
This is weird. I've attached the javac warnings it says are new and they have
nothing to do with this patch. test-patch must be broken in this regard. I
believe the patch is ready to go.
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
>Assignee: Jakob Homan
> Attachments: HADOOP-6080.patch, javac_warnings_diff.txt
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-6080:
Status: Patch Available (was: Open)
submitting patch.
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
> Assignee: Jakob Homan
> Attachments: HADOOP-6080.patch
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6080) Handling of Trash with quota
[
https://issues.apache.org/jira/browse/HADOOP-6080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HADOOP-6080:
Attachment: HADOOP-6080.patch
Patch adds a new option to the fsshell rm and rmr commands: -skipTrash, which
performs as expected. Adds to trash unit test to verify correct execution.
Changes documentation to reflect new option. Docs suggest this option as being
a solution when a directory is over quota.
Passes all commons unit tests. Running test patch now. Will post those
results when done.
> Handling of Trash with quota
> -
>
> Key: HADOOP-6080
> URL: https://issues.apache.org/jira/browse/HADOOP-6080
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
>Reporter: Koji Noguchi
> Assignee: Jakob Homan
> Attachments: HADOOP-6080.patch
>
>
> Currently with quota turned on, user cannot call '-rmr' on large directory
> that causes over quota.
> {noformat}
> [knoguchi src]$ hadoop dfs -rmr /tmp/net2
> rmr: Failed to move to trash: hdfs://abc.def.com/tmp/net2
> [knoguchi src]$ hadoop dfs -mv /tmp/net2 /user/knoguchi/.Trash/Current
> mv: org.apache.hadoop.hdfs.protocol.QuotaExceededException: The quota of
> /user/knoguchi is exceeded: namespace
> quota=37500 file count=37757, diskspace quota=-1 diskspace=1991250043353
> {noformat}
> Besides from error message being unfriendly, how should this be handled?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
