[jira] [Created] (HADOOP-8587) HarFileSystem access of harMetaCache isn't threadsafe
Eli Collins created HADOOP-8587: --- Summary: HarFileSystem access of harMetaCache isn't threadsafe Key: HADOOP-8587 URL: https://issues.apache.org/jira/browse/HADOOP-8587 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Eli Collins HarFileSystem's use of the static harMetaCache map is not threadsafe. Credit to Todd for pointing this out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8586) Fixup a bunch of SPNEGO misspellings
Eli Collins created HADOOP-8586: --- Summary: Fixup a bunch of SPNEGO misspellings Key: HADOOP-8586 URL: https://issues.apache.org/jira/browse/HADOOP-8586 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins SPNEGO is misspelled as "SPENGO" a bunch of places. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Jetty fixes for Hadoop
+1 from me too. We've had this in CDH since Sep '11 and been working much better than the stock 6.1.26. -Todd On Tue, Jul 10, 2012 at 3:14 PM, Owen O'Malley wrote: > On Tue, Jul 10, 2012 at 2:59 PM, Thomas Graves wrote: > >> I'm +1 for adding it. >> > > I'm +1 also. > > -- Owen -- Todd Lipcon Software Engineer, Cloudera
Re: Jetty fixes for Hadoop
On Tue, Jul 10, 2012 at 2:59 PM, Thomas Graves wrote: > I'm +1 for adding it. > I'm +1 also. -- Owen
Re: Jetty fixes for Hadoop
I'm +1 for adding it. Tom On 7/9/12 6:22 PM, "Suresh Srinivas" wrote: > Many folks running Apache Hadoop releases run into Jetty related issues. > Some fixes to Jetty alleviates this problem. See Todd's comment: See > https://issues.apache.org/jira/browse/MAPREDUCE-2980?focusedCommentId=13291112 > &page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment > -13291112 > > I propose making this available to Apache Hadoop by: > # Hosting the code in shared github repository and publishing the artifacts > built to maven central > # Update the Apache Hadoop to use the Jetty jars with fixes. > > Feedback welcome. > > Regards, > Suresh
Re: Problem setting up 1st generation Hadoop-0.20 (ANT build) in Eclipse
I tried *"ant tar"* but it requested a forrest home directory ,so I ran *"ant -Dforrest.home=/path/apache-forrest-0.8 compile-core tar"* but this gets stuck at an Exception "Exception in thread "main" java.lang.InternalError: Can't connect to X11 window server using ':0' as the value of the DISPLAY variable." No idea what this exception means. How come there isn't a good documentation or a BUILDING.txt file for MR1 releases ? Any help regarding this is appreciated.Thanks On Tue, Jul 10, 2012 at 4:29 PM, Harsh J wrote: > Hey Pavan, > > Try an "ant tar". For more ant targets, read the build.xml at the root > of your checkout. > > On Wed, Jul 11, 2012 at 1:15 AM, Pavan Kulkarni > wrote: > > Thanks a lot Harsh.I could set it up without any errors. > > It would be great if you could provide me any pointers on how to build a > > binary distribution tar file. > > The information on wiki and in BUILDING.txt only has Maven > > instructions.Thanks > > > > On Tue, Jul 10, 2012 at 2:39 PM, Harsh J wrote: > > > >> Hey Pavan, > >> > >> The 0.20.x version series was renamed recently to 1.x. Hence, you need > >> to use the branch-1 code path if you want the latest stable branch > >> (MR1, etc.) code. > >> > >> Do these once you have ant 1.8 and a Sun/Oracle JDK 1.6 installed, and > >> you should have it: > >> > >> $ git clone http://github.com/apache/hadoop-common.git hadoop-1 > >> $ cd hadoop-1; git checkout branch-1 > >> $ ant eclipse > >> > >> (Now export this directory into Eclipse as a Java project) > >> > >> HTH. > >> > >> On Wed, Jul 11, 2012 at 12:00 AM, Pavan Kulkarni > >> wrote: > >> > Hi all, > >> > > >> > I am trying to setup hadoop 1st generation 0.20 in Eclipse which > still > >> > uses Ant as its build tool. > >> > The build was successful , but when I want to set it up in the Eclipse > >> IDE > >> > i.e > >> > *File-> new Project-> Project from existing ANT build file -> Select > >> > build.xml ->Finish* > >> > I get this following error : > >> > *Problem setting the classpath of the project from the javac > classpath: > >> > Reference ivy-common.classpath not found.* > >> > > >> > I have tried finding solutions online but couldn't get a concrete > one. > >> Are > >> > there any sources or workarounds on setting 1st generation > >> > Hadoop in Eclipse.? > >> > > >> > Also my second question was how to build a binary tar file for > >> hadoop-0.20 > >> > which still uses ANT. The wiki pages only have information for maven. > >> > Any help is highly appreciated.Thanks > >> > -- > >> > > >> > --With Regards > >> > Pavan Kulkarni > >> > >> > >> > >> -- > >> Harsh J > >> > > > > > > > > -- > > > > --With Regards > > Pavan Kulkarni > > > > -- > Harsh J > -- --With Regards Pavan Kulkarni
[jira] [Resolved] (HADOOP-8584) test-patch.sh should not immediately exit when no tests are added or modified
[ https://issues.apache.org/jira/browse/HADOOP-8584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HADOOP-8584. - Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed I've committed this to trunk. > test-patch.sh should not immediately exit when no tests are added or modified > - > > Key: HADOOP-8584 > URL: https://issues.apache.org/jira/browse/HADOOP-8584 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 3.0.0 > > Attachments: HADOOP-8584.001.patch > > > test-patch.sh should not immediately exit when no tests are added or modified. > Although it's good to note whether or not a patch introduces or modifies > tests, it's not good to abort the Jenkins patch process if it did not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Problem setting up 1st generation Hadoop-0.20 (ANT build) in Eclipse
Hey Pavan, Try an "ant tar". For more ant targets, read the build.xml at the root of your checkout. On Wed, Jul 11, 2012 at 1:15 AM, Pavan Kulkarni wrote: > Thanks a lot Harsh.I could set it up without any errors. > It would be great if you could provide me any pointers on how to build a > binary distribution tar file. > The information on wiki and in BUILDING.txt only has Maven > instructions.Thanks > > On Tue, Jul 10, 2012 at 2:39 PM, Harsh J wrote: > >> Hey Pavan, >> >> The 0.20.x version series was renamed recently to 1.x. Hence, you need >> to use the branch-1 code path if you want the latest stable branch >> (MR1, etc.) code. >> >> Do these once you have ant 1.8 and a Sun/Oracle JDK 1.6 installed, and >> you should have it: >> >> $ git clone http://github.com/apache/hadoop-common.git hadoop-1 >> $ cd hadoop-1; git checkout branch-1 >> $ ant eclipse >> >> (Now export this directory into Eclipse as a Java project) >> >> HTH. >> >> On Wed, Jul 11, 2012 at 12:00 AM, Pavan Kulkarni >> wrote: >> > Hi all, >> > >> > I am trying to setup hadoop 1st generation 0.20 in Eclipse which still >> > uses Ant as its build tool. >> > The build was successful , but when I want to set it up in the Eclipse >> IDE >> > i.e >> > *File-> new Project-> Project from existing ANT build file -> Select >> > build.xml ->Finish* >> > I get this following error : >> > *Problem setting the classpath of the project from the javac classpath: >> > Reference ivy-common.classpath not found.* >> > >> > I have tried finding solutions online but couldn't get a concrete one. >> Are >> > there any sources or workarounds on setting 1st generation >> > Hadoop in Eclipse.? >> > >> > Also my second question was how to build a binary tar file for >> hadoop-0.20 >> > which still uses ANT. The wiki pages only have information for maven. >> > Any help is highly appreciated.Thanks >> > -- >> > >> > --With Regards >> > Pavan Kulkarni >> >> >> >> -- >> Harsh J >> > > > > -- > > --With Regards > Pavan Kulkarni -- Harsh J
Re: Problem setting up 1st generation Hadoop-0.20 (ANT build) in Eclipse
Thanks a lot Harsh.I could set it up without any errors. It would be great if you could provide me any pointers on how to build a binary distribution tar file. The information on wiki and in BUILDING.txt only has Maven instructions.Thanks On Tue, Jul 10, 2012 at 2:39 PM, Harsh J wrote: > Hey Pavan, > > The 0.20.x version series was renamed recently to 1.x. Hence, you need > to use the branch-1 code path if you want the latest stable branch > (MR1, etc.) code. > > Do these once you have ant 1.8 and a Sun/Oracle JDK 1.6 installed, and > you should have it: > > $ git clone http://github.com/apache/hadoop-common.git hadoop-1 > $ cd hadoop-1; git checkout branch-1 > $ ant eclipse > > (Now export this directory into Eclipse as a Java project) > > HTH. > > On Wed, Jul 11, 2012 at 12:00 AM, Pavan Kulkarni > wrote: > > Hi all, > > > > I am trying to setup hadoop 1st generation 0.20 in Eclipse which still > > uses Ant as its build tool. > > The build was successful , but when I want to set it up in the Eclipse > IDE > > i.e > > *File-> new Project-> Project from existing ANT build file -> Select > > build.xml ->Finish* > > I get this following error : > > *Problem setting the classpath of the project from the javac classpath: > > Reference ivy-common.classpath not found.* > > > > I have tried finding solutions online but couldn't get a concrete one. > Are > > there any sources or workarounds on setting 1st generation > > Hadoop in Eclipse.? > > > > Also my second question was how to build a binary tar file for > hadoop-0.20 > > which still uses ANT. The wiki pages only have information for maven. > > Any help is highly appreciated.Thanks > > -- > > > > --With Regards > > Pavan Kulkarni > > > > -- > Harsh J > -- --With Regards Pavan Kulkarni
[jira] [Created] (HADOOP-8584) test-patch.sh should not immediately exit when no tests are added or modified
Colin Patrick McCabe created HADOOP-8584: Summary: test-patch.sh should not immediately exit when no tests are added or modified Key: HADOOP-8584 URL: https://issues.apache.org/jira/browse/HADOOP-8584 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe test-patch.sh should not immediately exit when no tests are added or modified. Although it's good to note whether or not a patch introduces or modifies tests, it's not good to abort the Jenkins patch process if it did not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Problem setting up 1st generation Hadoop-0.20 (ANT build) in Eclipse
Hey Pavan, The 0.20.x version series was renamed recently to 1.x. Hence, you need to use the branch-1 code path if you want the latest stable branch (MR1, etc.) code. Do these once you have ant 1.8 and a Sun/Oracle JDK 1.6 installed, and you should have it: $ git clone http://github.com/apache/hadoop-common.git hadoop-1 $ cd hadoop-1; git checkout branch-1 $ ant eclipse (Now export this directory into Eclipse as a Java project) HTH. On Wed, Jul 11, 2012 at 12:00 AM, Pavan Kulkarni wrote: > Hi all, > > I am trying to setup hadoop 1st generation 0.20 in Eclipse which still > uses Ant as its build tool. > The build was successful , but when I want to set it up in the Eclipse IDE > i.e > *File-> new Project-> Project from existing ANT build file -> Select > build.xml ->Finish* > I get this following error : > *Problem setting the classpath of the project from the javac classpath: > Reference ivy-common.classpath not found.* > > I have tried finding solutions online but couldn't get a concrete one. Are > there any sources or workarounds on setting 1st generation > Hadoop in Eclipse.? > > Also my second question was how to build a binary tar file for hadoop-0.20 > which still uses ANT. The wiki pages only have information for maven. > Any help is highly appreciated.Thanks > -- > > --With Regards > Pavan Kulkarni -- Harsh J
Problem setting up 1st generation Hadoop-0.20 (ANT build) in Eclipse
Hi all, I am trying to setup hadoop 1st generation 0.20 in Eclipse which still uses Ant as its build tool. The build was successful , but when I want to set it up in the Eclipse IDE i.e *File-> new Project-> Project from existing ANT build file -> Select build.xml ->Finish* I get this following error : *Problem setting the classpath of the project from the javac classpath: Reference ivy-common.classpath not found.* I have tried finding solutions online but couldn't get a concrete one. Are there any sources or workarounds on setting 1st generation Hadoop in Eclipse.? Also my second question was how to build a binary tar file for hadoop-0.20 which still uses ANT. The wiki pages only have information for maven. Any help is highly appreciated.Thanks -- --With Regards Pavan Kulkarni
[jira] [Created] (HADOOP-8583) Globbing is not correctly handled in a few cases on Windows
Ramya Sunil created HADOOP-8583:
---
Summary: Globbing is not correctly handled in a few cases on
Windows
Key: HADOOP-8583
URL: https://issues.apache.org/jira/browse/HADOOP-8583
Project: Hadoop Common
Issue Type: Bug
Environment: Windows
Reporter: Ramya Sunil
Glob handling fails in a few cases on a Windows environment.
For example:
{noformat}
c:\> hadoop dfs -ls /
Found 2 items
drwxrwxrwx - Administrator supergroup 0 2012-07-06 15:00 /tmp
drwxr-xr-x - Administrator supergroup 0 2012-07-06 18:52 /user
c:\> hadoop dfs -ls /tmpInvalid*
Found 2 items
drwxr-xr-x - Administrator supergroup 0 2012-07-10 18:50
/user/Administrator/sortInputDir
drwxr-xr-x - Administrator supergroup 0 2012-07-10 18:50
/user/Administrator/sortOutputDir
c:\> hadoop dfs -rmr /tmp/*
Usage: java FsShell [-rmr [-skipTrash] ]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hadoop API performance testcases
Hi all, are there any performance testcases for hadoop APIs? We are looking for testcases to time performance of each API. Best Regards Amir Sanjar
[jira] [Resolved] (HADOOP-6817) SequenceFile.Reader can't read gzip format compressed sequence file which produce by a mapreduce job without native compression library
[
https://issues.apache.org/jira/browse/HADOOP-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J resolved HADOOP-6817.
-
Resolution: Duplicate
This is being addressed via HADOOP-8582.
> SequenceFile.Reader can't read gzip format compressed sequence file which
> produce by a mapreduce job without native compression library
> ---
>
> Key: HADOOP-6817
> URL: https://issues.apache.org/jira/browse/HADOOP-6817
> Project: Hadoop Common
> Issue Type: Bug
> Components: io
>Affects Versions: 0.20.2
> Environment: Cluster:CentOS 5,jdk1.6.0_20
> Client:Mac SnowLeopard,jdk1.6.0_20
>Reporter: Wenjun Huang
>
> An hadoop job output a gzip compressed sequence file(whether record
> compressed or block compressed).The client program use SequenceFile.Reader to
> read this sequence file,when reading the client program shows the following
> exceptions:
> 2090 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2091 [main] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new
> decompressor
> Exception in thread "main" java.io.EOFException
> at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:207)
> at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:197)
> at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:136)
> at java.util.zip.GZIPInputStream.(GZIPInputStream.java:58)
> at java.util.zip.GZIPInputStream.(GZIPInputStream.java:68)
> at
> org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.(GzipCodec.java:92)
> at
> org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.(GzipCodec.java:101)
> at
> org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:170)
> at
> org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:180)
> at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1520)
> at
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1428)
> at
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1417)
> at
> org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1412)
> at
> com.shiningware.intelligenceonline.taobao.mapreduce.HtmlContentSeqOutputView.main(HtmlContentSeqOutputView.java:28)
> I studied the code in org.apache.hadoop.io.SequenceFile.Reader.init method
> and read:
> // Initialize... *not* if this we are constructing a temporary Reader
> if (!tempReader) {
> valBuffer = new DataInputBuffer();
> if (decompress) {
> valDecompressor = CodecPool.getDecompressor(codec);
> valInFilter = codec.createInputStream(valBuffer, valDecompressor);
> valIn = new DataInputStream(valInFilter);
> } else {
> valIn = valBuffer;
> }
> the problem seems to be caused by "valBuffer = new DataInputBuffer();"
> ,because GzipCodec.createInputStream creates an instance of GzipInputStream
> whose constructor creates an instance of ResetableGZIPInputStream class.When
> ResetableGZIPInputStream's constructor calls it base class
> java.util.zip.GZIPInputStream's constructor ,it trys to read the empty
> "valBuffer = new DataInputBuffer();" and get no content,so it throws an
> EOFException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-5555) JobClient should provide an API to return the job names of jobs
[ https://issues.apache.org/jira/browse/HADOOP-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HADOOP-. - Resolution: Not A Problem The JobClient provides both Job and RunningJob returns via some of its cluster-connecting methods, that in turn provide an API to retrieve the Job Name string already. Hence, this has already been fixed. For the 'hadoop job -list' enhancement to show the same, see MAPREDUCE-4424 instead (which I just forked out). Resolving as Not a Problem (anymore). > JobClient should provide an API to return the job names of jobs > --- > > Key: HADOOP- > URL: https://issues.apache.org/jira/browse/HADOOP- > Project: Hadoop Common > Issue Type: Improvement >Reporter: Runping Qi > > Currently, there seems to be no way to get the job name of a job from its job > id. > The JobClient should provide a way to do so. > Also, the command line "hadoop job -list" should also return the job names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-5754) Add link to training from website
[ https://issues.apache.org/jira/browse/HADOOP-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved HADOOP-5754. --- Resolution: Fixed > Add link to training from website > - > > Key: HADOOP-5754 > URL: https://issues.apache.org/jira/browse/HADOOP-5754 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Tom White >Assignee: Tom White > Fix For: site > > Attachments: HADOOP-5754.patch, hadoop-5754-v2.patch, > hadoop-5754-v3.patch, hadoop-5754.patch > > > Add a link to training videos from the getting started section (like Pig, see > http://hadoop.apache.org/pig/). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8582) Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries.
Paul Wilkinson created HADOOP-8582: -- Summary: Improve error reporting for GZIP-compressed SequenceFiles with missing native libraries. Key: HADOOP-8582 URL: https://issues.apache.org/jira/browse/HADOOP-8582 Project: Hadoop Common Issue Type: Improvement Components: io Environment: Centos 5.8, Java 6 Update 26 Reporter: Paul Wilkinson Priority: Minor At present it is not possible to write or read block-compressed SequenceFiles using the GZIP codec without the native libraries being available. The SequenceFile.Writer code checks for the availability of native libraries and throws a useful exception, but the SequenceFile.Reader doesn't do the same: Exception in thread "main" java.io.EOFException at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:249) at java.util.zip.GZIPInputStream.readUShort(GZIPInputStream.java:239) at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:142) at java.util.zip.GZIPInputStream.(GZIPInputStream.java:58) at java.util.zip.GZIPInputStream.(GZIPInputStream.java:67) at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream$ResetableGZIPInputStream.(GzipCodec.java:95) at org.apache.hadoop.io.compress.GzipCodec$GzipInputStream.(GzipCodec.java:104) at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:173) at org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java:183) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1591) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1493) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1480) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1475) at test.SequenceReader.read(SequenceReader.java:23) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
