[jira] [Resolved] (HADOOP-3126) org.apache.hadoop.examples.RandomTextWriter$Counters fluctuate when RandonTextWriter job is running
[ https://issues.apache.org/jira/browse/HADOOP-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3126. -- Resolution: Fixed I'm going to close this as fixed since it probably was. > org.apache.hadoop.examples.RandomTextWriter$Counters fluctuate when > RandonTextWriter job is running > --- > > Key: HADOOP-3126 > URL: https://issues.apache.org/jira/browse/HADOOP-3126 > Project: Hadoop Common > Issue Type: Bug >Reporter: Runping Qi > > On the web GUI page, the value for RECORDS_WRITTEN and BYTES_WRITTEN do not > increase monotonically. > Rather, their values go up and down. > I suspect something wrong with how the counters are updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3148) build-contrib.xml should inherit hadoop version parameter from root build.xml
[
https://issues.apache.org/jira/browse/HADOOP-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-3148.
--
Resolution: Fixed
Stale issue.
> build-contrib.xml should inherit hadoop version parameter from root build.xml
> -
>
> Key: HADOOP-3148
> URL: https://issues.apache.org/jira/browse/HADOOP-3148
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Vinod Kumar Vavilapalli
>Priority: Minor
>
> This is needed in HOD (and may be useful in other contrib projects), which,
> in some cases, may be compiled and built separately. After HADOOP-3137, HOD
> will obtain its version from build parameter ${version}, and this will fail
> to give proper version when built independently(at src/contrib/hod level).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-3262) make Hadoop compile under Apache Harmony
[ https://issues.apache.org/jira/browse/HADOOP-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3262. -- Resolution: Won't Fix RIP Harmony. > make Hadoop compile under Apache Harmony > > > Key: HADOOP-3262 > URL: https://issues.apache.org/jira/browse/HADOOP-3262 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Doug Cutting > Attachments: HADOOP-3262.patch > > > Some small changes are required to get Hadoop Core to compile with Apache > Harmony. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-145) io.skip.checksum.errors property clashes with LocalFileSystem#reportChecksumFailure
[ https://issues.apache.org/jira/browse/HADOOP-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-145. - Resolution: Fixed Long ago. > io.skip.checksum.errors property clashes with > LocalFileSystem#reportChecksumFailure > --- > > Key: HADOOP-145 > URL: https://issues.apache.org/jira/browse/HADOOP-145 > Project: Hadoop Common > Issue Type: Bug > Components: io >Reporter: stack >Assignee: Owen O'Malley > > Below is from email to the dev list on Tue, 11 Apr 2006 14:46:09 -0700. > Checksum errors seem to be a fact of life given the hardware we use. They'll > often cause my jobs to fail so I have been trying to figure how to just skip > the bad records and files. At the end is a note where Stefan pointed me at > 'io.skip.checksum.errors'. This property, when set, triggers special > handling of checksum errors inside SequenceFile$Reader: If a checksum, try to > skip to next record. Only, this behavior can conflict with another checksum > handler that moves aside the problematic file whenever a checksum error is > found. Below is from a recent log. > 060411 202203 task_r_22esh3 Moving bad file > /2/hadoop/tmp/task_r_22esh3/task_m_e3chga.out to > /2/bad_files/task_m_e3chga.out.1707416716 > 060411 202203 task_r_22esh3 Bad checksum at 3578152. Skipping entries. > 060411 202203 task_r_22esh3 Error running child > 060411 202203 task_r_22esh3 java.nio.channels.ClosedChannelException > 060411 202203 task_r_22esh3 at > sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:89) > 060411 202203 task_r_22esh3 at > sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:276) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.seek(LocalFileSystem.java:79) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.fs.FSDataInputStream$Checker.seek(FSDataInputStream.java:67) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.fs.FSDataInputStream$PositionCache.seek(FSDataInputStream.java:164) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:193) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:243) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.io.SequenceFile$Reader.seek(SequenceFile.java:420) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.io.SequenceFile$Reader.sync(SequenceFile.java:431) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.io.SequenceFile$Reader.handleChecksumException(SequenceFile.java:412) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:389) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:209) > 060411 202203 task_r_22esh3 at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:709) > (Ignore line numbers. My code is a little different from main because I've > other debugging code inside in SequenceFile. Otherwise I'm running w/ head > of hadoop). > The SequenceFile$Reader#handleChecksumException is trying to skip to next > record but the file has been closed by the move-aside. > On the list there is some discussion on merit of moving aside file when bad > checksum found. I've trying to test what happens if we leave the file in > place but haven't had a checksum error in a while. > Opening this issue so place to fill in experience as we go. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-353) Run datanode (or other hadoop servers) inside tomcat
[ https://issues.apache.org/jira/browse/HADOOP-353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-353. - Resolution: Won't Fix I'm closing this out as Won't Fix. At this point, I think there is zero interesting in making Hadoop run inside another app server. > Run datanode (or other hadoop servers) inside tomcat > > > Key: HADOOP-353 > URL: https://issues.apache.org/jira/browse/HADOOP-353 > Project: Hadoop Common > Issue Type: Improvement >Reporter: eric baldeschwieler > > Barry Kaplan is running hadoop data nodes inside tomcat and encountering some > issues. > http://issues.apache.org/jira/browse/HADOOP-211#action_12419360 > I'm filing this bug to capture discussion about the pros and cons of such an > approach. I'd be curious to know what others on the list (who know more > about java/tomcat) think about this proposal. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3228) Replace Text.hashcode with a better hash function for non-ascii strings
[ https://issues.apache.org/jira/browse/HADOOP-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3228. -- Resolution: Fixed I'm going to close this as fixed. > Replace Text.hashcode with a better hash function for non-ascii strings > --- > > Key: HADOOP-3228 > URL: https://issues.apache.org/jira/browse/HADOOP-3228 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Owen O'Malley > > Should we change the hash function for Text to something that handles > non-ascii characters better? > http://bailey.svn.sourceforge.net/viewvc/bailey/trunk/src/java/org/apache/bailey/util/Hash.java?view=markup -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3231) move Stringifier to org.apache.hadoop.io.serialization package and rename DefaultStringifier to Stringifier
[ https://issues.apache.org/jira/browse/HADOOP-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3231. -- Resolution: Incomplete closing this as stale. > move Stringifier to org.apache.hadoop.io.serialization package and rename > DefaultStringifier to Stringifier > --- > > Key: HADOOP-3231 > URL: https://issues.apache.org/jira/browse/HADOOP-3231 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Reporter: Owen O'Malley > > The Stringinfier already uses the Serializer under the hood, and thus doesn't > really need to be parameterized further. I propose that we move the > DefaultStringifier to org.apache.hadoop.io.serializer.Stringifier and remove > the Stringifier interface. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3249) Browsing DFS behind gateway
[ https://issues.apache.org/jira/browse/HADOOP-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3249. -- Resolution: Won't Fix Closing this as won't fix. Proxy solutions are totally realistic. Plus Apache Knox provides some of this functionality. > Browsing DFS behind gateway > --- > > Key: HADOOP-3249 > URL: https://issues.apache.org/jira/browse/HADOOP-3249 > Project: Hadoop Common > Issue Type: Wish > Environment: Red-Hat cluster >Reporter: Martin Boeker >Priority: Minor > Original Estimate: 5h > Remaining Estimate: 5h > > Dear Hadoop guys, > I'm urgently trying to make a way for users to be able to see the contents of > a Hadoop DFS that is behind a gateway. I'm using port forwarding on the > gateway itself to point to the DFS web interface, something like this: > [gateway_external_IP]:50070 >> [node_internal_IP]:50070 > This works fine, if I go to http://gateway_external_ip:50070/ I can view the > DFS cluster html page from the outside world. The problem is that if I click > on any of the slave node links, it forwards to http://node_hostname/.., which > obviously doesn't work. I really need to get this going, a couple of projects > require this to be implemented. > I'm willing to do this any way possible, I don't really need to use the 50070 > web interface, even a simple directory structure would do, but I'm not sure > how to implement that either, because I don't know of a way to make an httpd > or ftpd use "bin/hadoop dfs -lsr /" as the root directory. I'd also be > willing to make people use a proxy server if that would fix my issue somehow.. > If anyone can help, I would greatly appreciate it, like I said it's kind of > urgent and I'm running out of ideas to try.. > Thanks a lot in advance, > -Martin -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-2681) NullPointerException in TaskRunner.java when system property "hadoop.log.dir" is not set
[ https://issues.apache.org/jira/browse/HADOOP-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-2681. -- Resolution: Duplicate Closing this as a duplicate of HADOOP-3438. > NullPointerException in TaskRunner.java when system property "hadoop.log.dir" > is not set > > > Key: HADOOP-2681 > URL: https://issues.apache.org/jira/browse/HADOOP-2681 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 0.15.2 >Reporter: Xu Zhang >Priority: Minor > Labels: newbie > > Currently, NullPointerException exception is thrown on line 321 in > TaskRunner.java when system property "hadoop.log.dir" is not set. Instead of > a NullPointerException exception, I expected a default value for > "hadoop.log.dir" to be used, or to see a more meaningful error message that > could have helped me figure out what was wrong (like, telling me that I > needed to set "hadoop.log.dir" and how to do so). > Here is one instance of such exceptions: > WARN mapred.TaskRunner: task_200801181719_0001_m_00_0 Child Error > java.lang.NullPointerException > at java.io.File.(File.java:222) > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:321) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3437) mapred.job.tracker default value/docs appear out of sync with code
[ https://issues.apache.org/jira/browse/HADOOP-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3437. -- Resolution: Incomplete I'm going to close this as stale. > mapred.job.tracker default value/docs appear out of sync with code > -- > > Key: HADOOP-3437 > URL: https://issues.apache.org/jira/browse/HADOOP-3437 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 0.18.0 >Reporter: Steve Loughran >Priority: Minor > > This is the settings for mapred.job.tracker in SVN_HEAD. If you dont override > these, you get a RuntimeException > > mapred.job.tracker > local > The host and port that the MapReduce job tracker runs > at. If "local", then jobs are run in-process as a single map > and reduce task. > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3470) Bad coding style: The member fields in org.apache.hadoop.ipc.metrics.RpcMetrics are public
[
https://issues.apache.org/jira/browse/HADOOP-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-3470.
--
Resolution: Won't Fix
Closing this as won't fix given the metrics system has been replaced.
> Bad coding style: The member fields in
> org.apache.hadoop.ipc.metrics.RpcMetrics are public
> --
>
> Key: HADOOP-3470
> URL: https://issues.apache.org/jira/browse/HADOOP-3470
> Project: Hadoop Common
> Issue Type: Improvement
> Components: metrics
>Reporter: Tsz Wo Nicholas Sze
>
> In org.apache.hadoop.ipc.metrics.RpcMetrics,
> {code}
> //the following are member fields
> public MetricsTimeVaryingRate rpcQueueTime = new
> MetricsTimeVaryingRate("RpcQueueTime");
> public MetricsTimeVaryingRate rpcProcessingTime = new
> MetricsTimeVaryingRate("RpcProcessingTime");
> public Map metricsList =
> Collections.synchronizedMap(new HashMap());
> {code}
> Then, the fields are accessed directly in other classes. For example,
> org.apache.hadoop.ipc.RPC.Server.call(...)
> {code}
> ...
> MetricsTimeVaryingRate m =
> rpcMetrics.metricsList.get(call.getMethodName());
> if (m != null) {
> m.inc(processingTime);
> }
> else {
> rpcMetrics.metricsList.put(call.getMethodName(), new
> MetricsTimeVaryingRate(call.getMethodName()));
> m = rpcMetrics.metricsList.get(call.getMethodName());
> m.inc(processingTime);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-8568) DNS#reverseDns fails on IPv6 addresses
[
https://issues.apache.org/jira/browse/HADOOP-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-8568.
--
Resolution: Duplicate
I'm closing this JIRA out in favor of HADOOP-3619 since it has a more modern
patch associated with it.
> DNS#reverseDns fails on IPv6 addresses
> --
>
> Key: HADOOP-8568
> URL: https://issues.apache.org/jira/browse/HADOOP-8568
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Eli Collins
>Assignee: Tony Kew
> Labels: newbie
> Attachments: HADOOP-8568.patch
>
>
> DNS#reverseDns assumes hostIp is a v4 address (4 parts separated by dots),
> blows up if given a v6 address:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
> at org.apache.hadoop.net.DNS.reverseDns(DNS.java:79)
> at org.apache.hadoop.net.DNS.getHosts(DNS.java:237)
> at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:340)
> at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:358)
> at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:337)
> at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:235)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at
> org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1649)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-3632) Fix speculative execution or allow premature stop
[ https://issues.apache.org/jira/browse/HADOOP-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3632. -- Resolution: Incomplete Spec exec has changed a lot. Closing as stale. > Fix speculative execution or allow premature stop > - > > Key: HADOOP-3632 > URL: https://issues.apache.org/jira/browse/HADOOP-3632 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 0.16.3 >Reporter: Severin Hacker > Original Estimate: 72h > Remaining Estimate: 72h > > I run 50 iterations of a program with 500 maps and no reduces. I have noticed > the following behaviour: > In 50% of the iterations: > 499 maps finish in 50 seconds > 1 map finishes after 4 minutes > Total time is 4 minutes. > In 50% of the iterations: > 500 maps finish in 50 seconds > Total time is 50 seconds. > It would be nice if I could tell hadoop to stop after 99% of the maps have > finished (and not wait for that last straggler). In my application it's > perfectly fine if I only get 99% of the results, as long as the straggler is > not always using the same data. > Please fix! -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3634) Create tests for Hadoop metrics
[ https://issues.apache.org/jira/browse/HADOOP-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3634. -- Resolution: Fixed > Create tests for Hadoop metrics > --- > > Key: HADOOP-3634 > URL: https://issues.apache.org/jira/browse/HADOOP-3634 > Project: Hadoop Common > Issue Type: New Feature > Components: metrics >Affects Versions: 0.19.0 >Reporter: Lohit Vijayarenu > > It would be good to have a test case for hadoop metrics. We could use > FileContext or derive something out of NullContext to check the values > returned via metrics are correct. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3890) distcp: PathFilter for source files
[ https://issues.apache.org/jira/browse/HADOOP-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3890. -- Resolution: Incomplete Closing since distcpv1 has been replaced! > distcp: PathFilter for source files > --- > > Key: HADOOP-3890 > URL: https://issues.apache.org/jira/browse/HADOOP-3890 > Project: Hadoop Common > Issue Type: New Feature > Components: util >Reporter: Koji Noguchi >Priority: Minor > Attachments: H-3890-1.patch > > > I'd like distcp to be able to skip _logs/_temporary directories and files. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3753) metrics: FileContext support overwrite mode
[ https://issues.apache.org/jira/browse/HADOOP-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3753. -- Resolution: Incomplete Closing as stale. > metrics: FileContext support overwrite mode > --- > > Key: HADOOP-3753 > URL: https://issues.apache.org/jira/browse/HADOOP-3753 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Reporter: Craig Macdonald >Priority: Minor > Attachments: HADOOP-3753.v1.patch > > > FileContext currently continually appends to the metrics log file(s), > generating an ever lengthening file. > In some scenarios, it would be useful to simply write the current statistics > to the file once every period, then overwrite the file for the next period. > For instance, this could be useful if an external application parsed the > metrics output - e.g. Cacti to create realtime graphs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3834) Checkin the design document for HDFS appends into source control repository
[ https://issues.apache.org/jira/browse/HADOOP-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3834. -- Resolution: Won't Fix > Checkin the design document for HDFS appends into source control repository > --- > > Key: HADOOP-3834 > URL: https://issues.apache.org/jira/browse/HADOOP-3834 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Reporter: dhruba borthakur > > The design document for HDFS needs to be converted into forrest and checked > into repository. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3838) Add hdfs random-reading tests to gridmix
[ https://issues.apache.org/jira/browse/HADOOP-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3838. -- Resolution: Incomplete gridmix? who uses that anymore? > Add hdfs random-reading tests to gridmix > > > Key: HADOOP-3838 > URL: https://issues.apache.org/jira/browse/HADOOP-3838 > Project: Hadoop Common > Issue Type: Improvement >Reporter: stack > Attachments: gridmix-randread_v1.patch > > > Gridmix needs a nice-little random read test so we can track how hdfs is > improving over time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3839) hadoop should handle no cygwin on windows more gracefully
[ https://issues.apache.org/jira/browse/HADOOP-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3839. -- Resolution: Won't Fix With Microsoft's help, Hadoop now uses batch files on Windows. Won't fix! > hadoop should handle no cygwin on windows more gracefully > - > > Key: HADOOP-3839 > URL: https://issues.apache.org/jira/browse/HADOOP-3839 > Project: Hadoop Common > Issue Type: Improvement > Environment: Windows XP, cygwin not installed >Reporter: Steve Loughran >Priority: Minor > > There have been a couple of postings to hadoop core-user in which people > can't get hdfs to come up on windows because whoami isnt on the path, which > fails with an IOException error code 2. > To people not experienced in DOS error codes, this is a fairly meanless > number which invariably leads to time wasted and support emails. > 1. the error could be caught and handled by printing some better hints (point > to a wiki page?) > 2. is whoami really needed on DOS-based filesystems? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3862) Create network topology plugin that uses a configured rack ip mask
[ https://issues.apache.org/jira/browse/HADOOP-3862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3862. -- Resolution: Duplicate Closing this as a dupe of HADOOP-3625, which actually has a patch on it! > Create network topology plugin that uses a configured rack ip mask > -- > > Key: HADOOP-3862 > URL: https://issues.apache.org/jira/browse/HADOOP-3862 > Project: Hadoop Common > Issue Type: New Feature > Components: net >Reporter: Owen O'Malley > > We should have a Java class that answers network topology questions by > implementing DNSToSwitchMapping by using the host's IP address and a mask > that defines the addresses that are on the same rack. Therefore, if your > racks are defined by using the top 3 octets, you'd configure ff.ff.ff.00. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3876) Hadoop Core should support source filesfor multiple schedulers
[ https://issues.apache.org/jira/browse/HADOOP-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3876. -- Resolution: Fixed Closing this, as Hadoop supports multiple schedulers now. > Hadoop Core should support source filesfor multiple schedulers > -- > > Key: HADOOP-3876 > URL: https://issues.apache.org/jira/browse/HADOOP-3876 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Vivek Ratan > > Besides the default JT scheduling algorithm, there is work going on with at > least two more schedulers (HADOOP-3445, HADOOP-3746). HADOOP-3412 makes it > easier to plug in new schedulers into the JT. Where do we place the source > files for various schedulers so that it's easy for users to choose their > scheduler of choice during deployment, and easy for developers to add in more > schedulers into the framework (without inundating it). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3881) IPC client doesnt time out if far end handler hangs
[ https://issues.apache.org/jira/browse/HADOOP-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3881. -- Resolution: Incomplete Closing this as stale. > IPC client doesnt time out if far end handler hangs > --- > > Key: HADOOP-3881 > URL: https://issues.apache.org/jira/browse/HADOOP-3881 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Affects Versions: 1.0.0 >Reporter: Steve Loughran >Priority: Minor > > This is what appears to be happening in some changes of mine that > (inadventently) blocked JobTracker: if the client can connect to the far end > and invoke an operation, the far end has forever to deal with the request: > the client blocks too. > Clearly the far end shouldn't do this; its a serious problem to address. but > should the client hang? Should it not time out after some specifiable time > and signal that the far end isn't processing requests in a timely manner? > (marked as minor as this shouldn't arise in day to day operation. but it > should be easy to create a mock object to simulate this, and timeouts are > considered useful in an IPC) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3902) Ordering of the output statistics in the report page (jobtracker-details for a job)
[ https://issues.apache.org/jira/browse/HADOOP-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3902. -- Resolution: Incomplete Closing this as stale. > Ordering of the output statistics in the report page (jobtracker-details for > a job) > --- > > Key: HADOOP-3902 > URL: https://issues.apache.org/jira/browse/HADOOP-3902 > Project: Hadoop Common > Issue Type: Bug > Components: util >Affects Versions: 0.18.0 > Environment: Fedora Core X86 (amazon ec2), JDK 1.6 >Reporter: Flo Leibert > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > The ordering of the job statistics in the jobdetails.jsp seems very > unintuitive - and not in sync with previous versions. > It seems as if the rows should be ordered by their respective function (maps, > combines, reduces). > Example: > Map-Reduce Framework > Reduce input groups 0 1,936 1,936 > Combine output records0 0 0 > Map input records 41,580,847 0 41,580,847 > Reduce output records 0 664,803,173 664,803,173 > Map output bytes 988,918,560 0 988,918,560 > Map input bytes 1,100,931,203 0 1,100,931,203 > Map output records41,580,847 0 41,580,847 > Combine input records 0 0 0 > Reduce input records 0 41,580,847 41,580,847 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3915) reducers hang, jobtracker loosing completely track of them.
[ https://issues.apache.org/jira/browse/HADOOP-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3915. -- Resolution: Incomplete Closing this as stale. > reducers hang, jobtracker loosing completely track of them. > --- > > Key: HADOOP-3915 > URL: https://issues.apache.org/jira/browse/HADOOP-3915 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.17.1 > Environment: EC2, Debian Etch (but not the ec2-contrib stuff) > streaming.jar >Reporter: Andreas Kostyrka > Attachments: > hadoop-hadoop-jobtracker-ec2-67-202-58-97.compute-1.amazonaws.com.log > > > I just noticed the following curious situation: > -) 18 of 22 reducers are waiting for 3 hours or so with 0.01MB/s and no > progress. > -) hadoop job -kill-task does not work on the ids shown > -) killing all reduce work tasks (the spawned Python processes, not java > TaskTracker$Child) gets completely ignored by the JobTracker, the jobtracker > shows them still as running. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3969) Provide Mechanism to optionally expose public org.apache.hadoop.util.Services APIs
[
https://issues.apache.org/jira/browse/HADOOP-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-3969.
--
Resolution: Incomplete
Having REST interfaces for everything would be great, but this JIRA isn't the
place to do it at this point.
> Provide Mechanism to optionally expose public org.apache.hadoop.util.Services
> APIs
> ---
>
> Key: HADOOP-3969
> URL: https://issues.apache.org/jira/browse/HADOOP-3969
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Pete Wyckoff
>
> Enhance manageability of Hadoop Services by providing Jute, Thrift, REST, ...
> APIs to select methods in the Service class (or potentially others).
> This will allow external tools written in a myriad of languages to query the
> state of Hadoop Servers and/or interact with them.
> This can be encapsulated in the Service class by defining a very simple
> interface and then optionally instantiating such an implementation provided
> in the Configuration.
> Some methods to be implemented include all the public methods in Service:
> {code}
> ping()
> isRunning()
> terminate()
> getServiceState()
> verifyServiceState()
> isTerminated(),
> {code}
> INTERFACE:
> {code}
> package org.apache.hadoop.util;
> public interface ExposeServiceAPIs {
>/**
>* @param service - the service whose APIs are to be exposed
> * @param serviceName - a symbolic name for the service
> * @param configuration - the hadoop configuration object
> **/
> public void initialize(Service service, String serviceName, Configuration
> conf) throws IOException,
> public boolean start();
> public boolean stop();
> } ;
> {code}
> Two straightforward implementations of this would be:
> 1. Servlet that exposes the APIs via REST
> 2. Thrift DDL of the service APIs and an implementation in Java + bindings in
> C++, Java, Perl, Python, Ruby, PHP
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-3999) Dynamic host configuration system (via node side plugins)
[ https://issues.apache.org/jira/browse/HADOOP-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-3999. -- Resolution: Incomplete Closing this as stale. Much of this functionality has since been added to YARN and HDFS. Holes are slowly being closed! > Dynamic host configuration system (via node side plugins) > - > > Key: HADOOP-3999 > URL: https://issues.apache.org/jira/browse/HADOOP-3999 > Project: Hadoop Common > Issue Type: Improvement > Components: benchmarks, conf, metrics > Environment: Any >Reporter: Kai Mosebach > Attachments: cloud_divide.jpg > > > The MapReduce paradigma is limited to run MapReduce jobs with the lowest > common factor of all nodes in the cluster. > On the one hand this is wanted (cloud computing, throw simple jobs in, > nevermind who does it) > On the other hand this is limiting the possibilities quite a lot, for > instance if you had data which could/needs to be fed to a 3rd party interface > like Mathlab, R, BioConductor you could solve a lot more jobs via hadoop. > Furthermore it could be interesting to know about the OS, the architecture, > the performance of the node in relation to the rest of the cluster. > (Performance ranking) > i.e. if i'd know about a sub cluster of very computing performant nodes or a > sub cluster of very fast disk-io nodes, the job tracker could select these > nodes regarding a so called job profile (i.e. my job is a heavy computing job > / heavy disk-io job), which can usually be estimated by a developer before. > To achieve this, node capabilities could be introduced and stored in the DFS, > giving you > a1.) basic information about each node (OS, ARCH) > a2.) more sophisticated infos (additional software, path to software, > version). > a3.) PKI collected about the node (disc-io, cpu power, memory) > a4.) network throughput to neighbor hosts, which might allow generating a > network performance map over the cluster > This would allow you to > b1.) generate jobs that have a profile (computing intensive, disk io > intensive, net io intensive) > b2.) generate jobs that have software dependencies (run on Linux only, run on > nodes with MathLab only) > b3.) generate a performance map of the cluster (sub clusters of fast disk > nodes, sub clusters of fast CPU nodes, network-speed-relation-map between > nodes) > From step b3) you could then even acquire statistical information which could > again be fed into the DFS Namenode to see if we could store data on fast disk > subclusters only (that might need to be a tool outside of hadoop core though) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4047) Metrics for connection cleanup by the ipc/Server: cleanupConnections()
[ https://issues.apache.org/jira/browse/HADOOP-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4047. -- Resolution: Fixed I think this has been implemented. I'm sure Koji will re-open if not. :D > Metrics for connection cleanup by the ipc/Server: cleanupConnections() > -- > > Key: HADOOP-4047 > URL: https://issues.apache.org/jira/browse/HADOOP-4047 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Reporter: Koji Noguchi >Priority: Minor > > Request for metrics that shows the number of idle ipc connections closed from > the Server side. > This metrics would have helped when debugging HADOOP-4040. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4034) Redundant deprecation warnings in hadoop logs
[ https://issues.apache.org/jira/browse/HADOOP-4034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4034. -- Resolution: Fixed > Redundant deprecation warnings in hadoop logs > - > > Key: HADOOP-4034 > URL: https://issues.apache.org/jira/browse/HADOOP-4034 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 0.18.0 >Reporter: Yoram Kulbak > Original Estimate: 1h > Remaining Estimate: 1h > > Warnings in the form of > "org.apache.hadoop.fs.FileSystem - "localhost:57367" is a deprecated > filesystem name. Use "hdfs://localhost:57367/" instead." > are frequently emitted into the hadoop log. The problem is that the frequency > of these warnings floods the logs and makes it difficult to discover real > issues. > A short investigation reveals that while FileSystem.getFileSysName(URI) > returns the file system name without the hdfs:// scheme part, the method > FileSystem.fixName(String) complains about it and appends the hdfs:// back. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4048) ipc.Client: Log when Server side closes the socket while request is still pending
[
https://issues.apache.org/jira/browse/HADOOP-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-4048.
--
Resolution: Fixed
> ipc.Client: Log when Server side closes the socket while request is still
> pending
> --
>
> Key: HADOOP-4048
> URL: https://issues.apache.org/jira/browse/HADOOP-4048
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
>Reporter: Koji Noguchi
>Priority: Minor
>
> ipc/Client.java
> {noformat}
> 316 } catch (EOFException eof) {
> 317 // This is what happens when the remote side goes down
> 318 }
> {noformat}
> Request to log when Server side closes the socket while some requests are
> still pending.
> This would have helped when debugging HADOOP-4040.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-4055) clean up UGI to support savetoConf
[
https://issues.apache.org/jira/browse/HADOOP-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-4055.
--
Resolution: Incomplete
We should really burn UGI to the ground. But that's a different jira.
Closing this as stale.
> clean up UGI to support savetoConf
> --
>
> Key: HADOOP-4055
> URL: https://issues.apache.org/jira/browse/HADOOP-4055
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
>Reporter: Owen O'Malley
>
> Currently clients have to use UnixUserGroupInfo instead of UserGroupInfo
> because saveToConf is only defined in UnixUGI. We should add the abstract
> saveToConf in UGI.
> {code}
> UserGroupInfo:
> public abstract void saveToConf(Configuration conf);
> {code}
> and the matching body in UnixUGI.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-4059) Add tests that try starting the various hadoop command line scripts
[ https://issues.apache.org/jira/browse/HADOOP-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4059. -- Resolution: Duplicate marking this as a duplicate of one of the followups to HADOOP-9902! > Add tests that try starting the various hadoop command line scripts > --- > > Key: HADOOP-4059 > URL: https://issues.apache.org/jira/browse/HADOOP-4059 > Project: Hadoop Common > Issue Type: Test > Components: test >Affects Versions: 0.19.0 >Reporter: Steve Loughran > Original Estimate: 72h > Remaining Estimate: 72h > > Hadoop has lots of tests that start clusters, but I don't see any that test > the command line scripts working. Which means that changes to the scripts and > the code behind them may not get picked up rapidly, and regressions won't get > picked up by Hudson and apportioned to the specific patches. > Propose: > * an abstract test case that can exec scripts on startup; wait for them to > finish, kill them if they take too long > * test cases for every service (namenode, tasktracker, datanode, etc) > * tests that try invalid commands, -help options > * tests that start the services, assert that they are live, and shut them down -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4165) test-patch script is showing questionable number of Javac warnings
[ https://issues.apache.org/jira/browse/HADOOP-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4165. -- Resolution: Fixed > test-patch script is showing questionable number of Javac warnings > -- > > Key: HADOOP-4165 > URL: https://issues.apache.org/jira/browse/HADOOP-4165 > Project: Hadoop Common > Issue Type: Test > Components: test >Affects Versions: 0.18.0 >Reporter: Ramya Sunil >Priority: Minor > > test-patch is recording 881 number of javac warnings when run on trunk and > 216 javac warnings when run on "ANY" patch. > This behavior was observed even on an empty patch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4196) Possible performance enhancement in Hadoop compress module
[ https://issues.apache.org/jira/browse/HADOOP-4196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4196. -- Resolution: Incomplete I believe the compression code has changed quite a bit since this was filed. Closing as stale. > Possible performance enhancement in Hadoop compress module > -- > > Key: HADOOP-4196 > URL: https://issues.apache.org/jira/browse/HADOOP-4196 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 0.18.0 >Reporter: Hong Tang > > There are several less performant implementation issues with the current > Hadoop compression module. Generally, the opportunities all come from the > fact that the granuarities of I/O operations from the CompressionStream and > DecompressionStream are not controllable by the users, and thus users are > forced to attach BufferedInputStream or BufferedOutputStream to both ends of > the CompressionStream and DecompressionStream: > - ZlibCompressor: always returns false from needInput() after setInput(), and > thus lead to a native call deflateBytesDirect() for almost every write() > operation from CompressorStream(). This becomes problematic when applications > call write() on the CompressorStream with small write sizes (e.g. one byte at > a time). It is better to follow similar code path in LzoCompressor and append > to internal uncompressed data buffer. > - CompressorStream: whenever the compressor produces some compressed data, it > will directly issue write() calls to the down stream. Could be improved by > keep appending to the byte[] until it is full (or half full) before writing > to the down stream. Otherwise, applications have to use a > BufferedOutputStream as the down stream in case the output sizes from > CompressorStream is too small. This generally causes double buffering. > - BlockCompressorStream: similar issue as described above. > - BlockDecompressorStream: getCompressedData() reads only one compressed > chunk at a time. Could be better to read a full buffer, and then obtain > compressed chunk from buffer (similar to DecompressStream is doing, but > admittedly a bit more complicated). > In generally, the following could be some guideline of > Compressor/Decompressor and CompressorStream/DecompressorStream > design/implementation that can give users some performance guarantee: > - Compressor and Decompressor keep two DirectByteBuffer, the size of which > should be tuned to be optimal with regard to the specific > compression/decompression algorithm. Ensure always call Compressor.compress() > will a full (or near full) uncompressed data DirectBuffer. > - CompressorStream and DecompressorStream maintains a byte[] to read data > from the down stream. The size of the byte[] should be user customizable (add > a bufferSize parameter to CompressionCodec's createInputStream and > createOutputStream interface). Ensure that I/O from the down stream at or > near the granularity of the size of the byte[]. So applications can simply > rely on the buffering inside CompressorStream and DecompressorStream (for the > case of LZO: BlockCompressorStream and BlockDecompressorStream). > A more radical change would be to let the downward InputStream to directly > deposit data to a ByteBuffer or the downard OutputStream to accept input data > from ByteBuffer. We may call it ByteBufferInputStream and > ByteBufferOutputStream. The CompressorStream and DecompressorStream may > simply test whether the down stream indeed implements such interfaces and > bypass its own byte[] buffer if true. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4201) [patch] native library build script uses unportable sed(1) regexp's
[ https://issues.apache.org/jira/browse/HADOOP-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4201. -- Resolution: Fixed > [patch] native library build script uses unportable sed(1) regexp's > --- > > Key: HADOOP-4201 > URL: https://issues.apache.org/jira/browse/HADOOP-4201 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 0.18.0 > Environment: FreeBSD 7.0-STABLE amd64 >Reporter: Ruslan Ermilov > Attachments: HADOOP-4201.patch, hadoop-0.18.0.patch > > > The native library build script uses unportable sed(1) regular expressions, > making it impossible to compile a library using non-GNU sed(1). > In particular, any POSIX-conformant sed(1) implementation will fail. The > following patch has been tested to work on both Linux (Ubuntu) and FreeBSD. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4208) Shouldn't delete the system dir after a JobTracker recovery
[ https://issues.apache.org/jira/browse/HADOOP-4208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4208. -- Resolution: Won't Fix JT recovery is just plain broken and at this point won't get fixed > Shouldn't delete the system dir after a JobTracker recovery > --- > > Key: HADOOP-4208 > URL: https://issues.apache.org/jira/browse/HADOOP-4208 > Project: Hadoop Common > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Amar Kamat > > Debugging JobTracker crashes will be easier if the files are preserved rather > than deleted on recovery, probably in a sub-directory. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4215) test-patch should have a mode where developers can turn on running core and contrib tests.
[ https://issues.apache.org/jira/browse/HADOOP-4215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4215. -- Resolution: Not a Problem > test-patch should have a mode where developers can turn on running core and > contrib tests. > -- > > Key: HADOOP-4215 > URL: https://issues.apache.org/jira/browse/HADOOP-4215 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Reporter: Hemanth Yamijala > > There are scenarios, such as on a feature freeze date, *smile*, when > developers rely on test-patch, rather than hudson to ensure their patches are > not causing regressions. For tests though, developers still have to run core > and contrib tests and grep on the output to detect failures, which can lead > to human errors. Having a mode where test-patch can optionally be asked to > run the core and contrib tests also and report on the consolidated status > would greatly help. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4243) Serialization framework use SequenceFile/TFile/Other metadata to instantiate deserializer
[ https://issues.apache.org/jira/browse/HADOOP-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4243. -- Resolution: Duplicate > Serialization framework use SequenceFile/TFile/Other metadata to instantiate > deserializer > - > > Key: HADOOP-4243 > URL: https://issues.apache.org/jira/browse/HADOOP-4243 > Project: Hadoop Common > Issue Type: Improvement > Components: contrib/serialization >Reporter: Pete Wyckoff > > SequenceFile metadata is useful for storing additional information about the > serialized data, for example, for RecordIO, whether the data is CSV or > Binary. For thrift, the same thing - Binary, JSON, ... > For Hive, this may be especially important, because it has a Dynamic generic > serializer/deserializer that takes its DDL at runtime (as opposed to RecordIO > and Thrift which require pre-compilation into a specific class whose name can > be stored in the sequence file key or value class). In this case, the class > name is like Record.java in RecordIO - it doesn't tell you anything without > the DDL. > One way to address this could be adding the sequence file metadata to the > getDeserializer call in Serialization interface. The api would then be > something like getDeserializer(Class, Map metadata) or > Properties metadata. > But, I am open to proposals. > This also means that saying a class implements Writable is not enough to > necessarily deserialize it since it may do specific actions based on the > metadata - e.g., RecordIO might determine whether to use CSV rather than the > default Binary deserialization. > There's the other issue of the getSerializer returning the metadata to be > written to the Sequence/T File. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4270) hive cli does not pass hadoop location correctly and does not work in correctly in local mode
[ https://issues.apache.org/jira/browse/HADOOP-4270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4270. -- Resolution: Not a Problem Remember when Hive was a sub-project? Yeah, me neither. > hive cli does not pass hadoop location correctly and does not work in > correctly in local mode > - > > Key: HADOOP-4270 > URL: https://issues.apache.org/jira/browse/HADOOP-4270 > Project: Hadoop Common > Issue Type: Bug >Reporter: Joydeep Sen Sarma > > the hadoop shell script location has to be set in conf variable > hadoop.bin.path. this is being passed in by hive shell script while invoking > the hive cli - but incorrect switch is being used as a result of which it is > not available when a subprocess is spawned for map-reduce. > only affect local mode since for normal mode we don't use sub-processes while > spawning jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4264) DFSIO is failing on 500 nodes cluster
[ https://issues.apache.org/jira/browse/HADOOP-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4264. -- Resolution: Fixed > DFSIO is failing on 500 nodes cluster > - > > Key: HADOOP-4264 > URL: https://issues.apache.org/jira/browse/HADOOP-4264 > Project: Hadoop Common > Issue Type: Bug > Components: io, test >Affects Versions: 0.19.0 >Reporter: Suman Sehgal > > On executing following command : > bin/hadoop jar ~/hadoop/hadoop-0.19.0-test.jar TestDFSIO -write -nrFiles 990 > -fileSize 320 > This error occurs: > 08/09/24 06:15:03 INFO mapred.JobClient: map 98% reduce 32% > java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1201) > at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:236) > at org.apache.hadoop.fs.TestDFSIO.writeTest(TestDFSIO.java:218) > at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:354) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) > at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) > at org.apache.hadoop.test.AllTestDriver.main(AllTestDriver.java:77) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:165) > at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) > On looking at hadoop logs, It seems that file names are clashing > 2008-09-24 06:21:41,618 INFO org.apache.hadoop.mapred.JobTracker: Removed > completed task 'attempt_200809240600_0005_m_000802_2_136048515' from > 'tracker_/client x.x.x.x:x' > 2008-09-24 06:21:41,627 INFO org.apache.hadoop.mapred.JobTracker: Adding task > 'attempt_200809240600_0005_m_000802_4_136048515' to tip > task_200809240600_0005_m_000802, for tracker 'tracker_/client > x.x.x.x:x' > 2008-09-24 06:21:41,627 INFO org.apache.hadoop.mapred.JobInProgress: Choosing > rack-local task task_200809240600_0005_m_000802 > 2008-09-24 06:21:41,724 INFO org.apache.hadoop.mapred.TaskInProgress: Error > from attempt_200809240600_0005_m_000900_2_136048515: > org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to > create file /benchmarks/TestDFSIO/io_data/test_io_20 for > DFSClient_attempt_200809240600_0005_m_000900_2_136048515 on client client > x.x.x.x, because this file is already being created by > DFSClient_attempt_200809240600_0005_m_000900_0_136048515 on client x.x.x.x -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4312) Checksum error during execution of unit tests on linux environment.
[ https://issues.apache.org/jira/browse/HADOOP-4312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4312. -- Resolution: Fixed > Checksum error during execution of unit tests on linux environment. > > > Key: HADOOP-4312 > URL: https://issues.apache.org/jira/browse/HADOOP-4312 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 0.18.1 >Reporter: Suman Sehgal > > Following unit tests are failing for 0.18.1_H4277_H4271 > org.apache.hadoop.fs.TestLocalFileSystem.testAppend > Error Message > Checksum error: > /xx/workspace/hadoop-0.18.1_H4277_H4271/build/test/data/append/f at 0 > Stacktrace > org.apache.hadoop.fs.ChecksumException: Checksum error: > /xx/workspace/hadoop-0.18.1_H4277_H4271/build/test/data/append/f at 0 > at > org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:277) > at > org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:242) > at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:190) > at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:159) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > org.apache.hadoop.fs.TestLocalFileSystem.readFile(TestLocalFileSystem.java:43) > at > org.apache.hadoop.fs.TestLocalFileSystem.testAppend(TestLocalFileSystem.java:173) > org.apache.hadoop.dfs.TestDFSShell.testPut > Error Message > Checksum error: /xx/workspace/hadoop-0.18.1_H4277_H4271/build/test/data/f2 at > 0 > Stacktrace > java.io.IOException: Checksum error: > /xx/workspace/hadoop-0.18.1_H4277_H4271/build/test/data/f2 at 0 > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:187) > at > org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1173) > at org.apache.hadoop.dfs.TestDFSShell.testPut(TestDFSShell.java:254) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4334) ObjectFile on top of TFile
[ https://issues.apache.org/jira/browse/HADOOP-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4334. -- Resolution: Fixed I think TFile is mostly deprecated at this point. Closing as won't fix. > ObjectFile on top of TFile > -- > > Key: HADOOP-4334 > URL: https://issues.apache.org/jira/browse/HADOOP-4334 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Amir Youssefi >Assignee: Amir Youssefi > Attachments: ObjectFile_1.patch > > > Problem: > We need to have Object (Serialization/Deserialization) support for TFile. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4322) Input/Output Format for TFile
[ https://issues.apache.org/jira/browse/HADOOP-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4322. -- Resolution: Won't Fix I think TFile is mostly deprecated at this point. Closing as won't fix. > Input/Output Format for TFile > - > > Key: HADOOP-4322 > URL: https://issues.apache.org/jira/browse/HADOOP-4322 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Amir Youssefi >Assignee: Amir Youssefi > Attachments: ObjectFileInputOutputFormat_1.patch > > > Input/Output Format for TFile -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4352) a job stays in running state forever, even though all the tasks completed a long time ago
[ https://issues.apache.org/jira/browse/HADOOP-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4352. -- Resolution: Cannot Reproduce > a job stays in running state forever, even though all the tasks completed a > long time ago > - > > Key: HADOOP-4352 > URL: https://issues.apache.org/jira/browse/HADOOP-4352 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.17.2 >Reporter: Runping Qi > Attachments: jobtracker_jstatck_trace.out > > > I encountered a job that stays in running state forever, even though all the > tasks completed a long time ago. > The last lines in the job tracker log complain that it cannot connect to the > namenode of the dfs, although the dfs namenode works fine at present time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4357) Rename all the FSXxx classes to FsXxx
[ https://issues.apache.org/jira/browse/HADOOP-4357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4357. -- Resolution: Incomplete > Rename all the FSXxx classes to FsXxx > - > > Key: HADOOP-4357 > URL: https://issues.apache.org/jira/browse/HADOOP-4357 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Reporter: Tsz Wo Nicholas Sze > > There are two naming conventions in Hadoop, FSXxx and FsXxx. We should > rename all the FSXxx classes to FsXxx. See also > http://issues.apache.org/jira/browse/HADOOP-4044?focusedCommentId=12637296#action_12637296 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4605) should run old version of unit tests to check back-compatibility
[ https://issues.apache.org/jira/browse/HADOOP-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4605. -- Resolution: Not a Problem I'm going to close this as Not a Problem. With API classification, new behavior is old behavior for Stable and unit tests shouldn't to change. > should run old version of unit tests to check back-compatibility > > > Key: HADOOP-4605 > URL: https://issues.apache.org/jira/browse/HADOOP-4605 > Project: Hadoop Common > Issue Type: Improvement > Components: test >Reporter: Doug Cutting > > We should test back-compatibility by running unit tests from a prior release. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4611) Documentation for Tool interface is a bit busted
[ https://issues.apache.org/jira/browse/HADOOP-4611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4611. -- Resolution: Not a Problem Tool is still broken, but for different reasons. I'll file a different jira for that. > Documentation for Tool interface is a bit busted > > > Key: HADOOP-4611 > URL: https://issues.apache.org/jira/browse/HADOOP-4611 > Project: Hadoop Common > Issue Type: Bug > Components: documentation, util >Reporter: Jeff Hammerbacher >Priority: Minor > Labels: newbie > > The documentation for the Tool interface will not work out of the box. It > seems to have taken the Sort() implementation in examples, but has ripped out > some important information. > 1) args[1] and args[2] should probably be args[0] and args[1], as most > MapReduce tasks don't take the first argument that examples.jar takes > 2) int run() needs to actually return an int > 3) JobConf.setInputPath() and JobConf.setOutputPath() are deprecated. > 4) the call to ToolRunner.run() in main() should take "new MyApp()" instead > of "Sort()" as an argument > More generally, a working implementation of Tool in the docs would be handy. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10864) Tool documentenation is broken
Allen Wittenauer created HADOOP-10864: - Summary: Tool documentenation is broken Key: HADOOP-10864 URL: https://issues.apache.org/jira/browse/HADOOP-10864 Project: Hadoop Common Issue Type: Bug Components: documentation Reporter: Allen Wittenauer Priority: Minor Looking at http://hadoop.apache.org/docs/r2.4.1/api/org/apache/hadoop/util/Tool.html, at least one of the links is non-existent. There are likely other bugs in this documentation too. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4637) Unhandled failures starting jobs with S3 as backing store
[ https://issues.apache.org/jira/browse/HADOOP-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4637. -- Resolution: Incomplete Closing this as a stale issue. > Unhandled failures starting jobs with S3 as backing store > - > > Key: HADOOP-4637 > URL: https://issues.apache.org/jira/browse/HADOOP-4637 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 0.18.1 >Reporter: Robert > > I run Hadoop 0.18.1 on Amazon EC2, with S3 as the backing store. > When starting jobs, I sometimes get the following failure, which causes the > job to be abandoned: > org.apache.hadoop.ipc.RemoteException: java.io.IOException: > java.lang.NullPointerException > at > org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveBlock(Jets3tFileSystemStore.java:222) > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > at $Proxy4.retrieveBlock(Unknown Source) > at > org.apache.hadoop.fs.s3.S3InputStream.blockSeekTo(S3InputStream.java:160) > at org.apache.hadoop.fs.s3.S3InputStream.read(S3InputStream.java:119) > at java.io.DataInputStream.read(DataInputStream.java:83) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:47) > at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:214) > at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:150) > at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1212) > at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1193) > at org.apache.hadoop.mapred.JobInProgress.(JobInProgress.java:177) > at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1783) > at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) > at org.apache.hadoop.ipc.Client.call(Client.java:715) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) > at org.apache.hadoop.mapred.$Proxy5.submitJob(Unknown Source) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:788) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1026) > The stack trace suggests that copying the job file fails, because the HDFS S3 > filesystem can't find all of the expected block objects when it needs them. > Since S3 is an "eventually consistent" kind of a filesystem, and does not > always provide an up-to-date view of the stored data, this execution path > probably should be strengthened - at least to retry these failed operations, > or wait for the expected block file if it hasn't shown up yet. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-2830) Need to instrument Hadoop to get comprehensive network traffic metrics
[ https://issues.apache.org/jira/browse/HADOOP-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-2830. -- Resolution: Duplicate I'm pushing this into 289 too. > Need to instrument Hadoop to get comprehensive network traffic metrics > -- > > Key: HADOOP-2830 > URL: https://issues.apache.org/jira/browse/HADOOP-2830 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Runping Qi > > One of most often asked question regarding Hadoop performance is: was the job > cpu bounded, or disk bounded, or network bounded. > The first two parts can be answered based on metric data of individual > machines, thus are relatively easy to answer. > The third part is much harder, especially for a large cluster. To unswer the > question, we need to know the followings: > 1. The network traffic to and from the nodes in the cluster > 2. The network traffic going between node pairs through the switch they share > 3. The network traffic going through the back links between the switches > With these data, we can get a better insight on the relationship between > network bandwidth and hadoop performance. > We need to instrument the Hadoop code to obtain the above data. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4414) Remove stale references to dfs.client.buffer.dir
[ https://issues.apache.org/jira/browse/HADOOP-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4414. -- Resolution: Fixed Already fixed. > Remove stale references to dfs.client.buffer.dir > > > Key: HADOOP-4414 > URL: https://issues.apache.org/jira/browse/HADOOP-4414 > Project: Hadoop Common > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Priority: Minor > Labels: newbie > > These, I found while removing dfs.client.buffer.dir generation by HOD > (HADOOP-3814). I see other references to the parameter - in comments of > Configuration.java (+100), LocalDirAllocator.java(+66) and also generated in > the test case TestLocalDirAllocator.java. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-3163) FS.mkdirs()'s contract is unclear.
[
https://issues.apache.org/jira/browse/HADOOP-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-3163.
--
Resolution: Fixed
I'm going to close this out as fixed. Between HCFS and the new FileContext, I
think this issue is done.
> FS.mkdirs()'s contract is unclear.
> --
>
> Key: HADOOP-3163
> URL: https://issues.apache.org/jira/browse/HADOOP-3163
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Raghu Angadi
>
> {{FileSystem.mkdirs()}} returns a boolean but does not specify what a return
> value of true or false means. For e.g. HDFS returns never returns false
> (until HADOOP-3138).
> Should it be consistent with java.io.File.mkdirs()?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-6910) Replace or merge Configuration with Jakarta Commons Configuration
[ https://issues.apache.org/jira/browse/HADOOP-6910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6910. -- Resolution: Won't Fix > Replace or merge Configuration with Jakarta Commons Configuration > - > > Key: HADOOP-6910 > URL: https://issues.apache.org/jira/browse/HADOOP-6910 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jeff Hammerbacher > > Jakarta Commons Configuration: http://commons.apache.org/configuration/ -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4503) ant jar when run for first time does not inclue version information
[ https://issues.apache.org/jira/browse/HADOOP-4503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4503. -- Resolution: Incomplete Closing as stale. > ant jar when run for first time does not inclue version information > --- > > Key: HADOOP-4503 > URL: https://issues.apache.org/jira/browse/HADOOP-4503 > Project: Hadoop Common > Issue Type: Bug > Components: build > Environment: linux builds and windows builds >Reporter: Sreekanth Ramakrishnan > Labels: newbie > Attachments: antfile, antfile1 > > > Ant jar when run for first time does not include version information. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4522) Capacity Scheduler needs to re-read its configuration
[ https://issues.apache.org/jira/browse/HADOOP-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4522. -- Resolution: Fixed Fixed a while back. > Capacity Scheduler needs to re-read its configuration > - > > Key: HADOOP-4522 > URL: https://issues.apache.org/jira/browse/HADOOP-4522 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Vivek Ratan > Attachments: 4522.1.patch > > > An external application (an Ops script, or some CLI-based tool) can change > the configuration of the Capacity Scheduler (change the capacities of various > queues, for example) by updating its config file. This application then needs > to tell the Capacity Scheduler that its config has changed, which causes the > Scheduler to re-read its configuration. It's possible that the Capacity > Scheduler may need to interact with external applications in other similar > ways. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4582) create-hadoop-image doesn't fail with expired Java binary URL
[ https://issues.apache.org/jira/browse/HADOOP-4582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4582. -- Resolution: Won't Fix We removed this stuff a long time ago. Closing. > create-hadoop-image doesn't fail with expired Java binary URL > - > > Key: HADOOP-4582 > URL: https://issues.apache.org/jira/browse/HADOOP-4582 > Project: Hadoop Common > Issue Type: Bug > Components: contrib/cloud >Affects Versions: 0.18.1 >Reporter: Karl Anderson >Priority: Minor > > Part of creating a Hadoop EC2 image involves putting the URL for the Java > binary into hadoop-ec2-env.sh. Ths URL is time-sensitive; a working URL will > eventually redirect to a HTML warning page. create-hadoop-image-remote does > not notice this, and will create, bundle, and register a non-working image, > which launch-cluster will launch, but on which the hadoop commands will not > work. > To fix, check the output status of the "sh java.bin" command in > create-hadoop-image-remote, die with that status, and check for that status > when create-hadoop-image-remote is run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4594) Monitoring Scripts for Nagios
[ https://issues.apache.org/jira/browse/HADOOP-4594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4594. -- Resolution: Won't Fix While I believe that such a collection would be valuable, it is no longer appropriate to ship these with Hadoop with the general death of contrib. It'd better for something like Apache Extras or maybe Big Top or ... Closing. > Monitoring Scripts for Nagios > - > > Key: HADOOP-4594 > URL: https://issues.apache.org/jira/browse/HADOOP-4594 > Project: Hadoop Common > Issue Type: Wish >Reporter: Edward Capriolo >Priority: Minor > Attachments: HADOOP-4594.patch > > > I would like to create a set of local via NRPE and remote check scripts that > can be shipped with the hadoop distribution and used to monitor Hadoop. I > already have completed the NRPE scripts. The second set of scripts would use > wget to read the output of the hadoop web interfaces. Do these already exist? > I guess these would fall under a new contrib project. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4603) Installation on Solaris needs additional PATH setting
[ https://issues.apache.org/jira/browse/HADOOP-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4603. -- Resolution: Won't Fix Now I really am going to close this as Won't Fix, especially since we finally removed the requirement for whoami. > Installation on Solaris needs additional PATH setting > - > > Key: HADOOP-4603 > URL: https://issues.apache.org/jira/browse/HADOOP-4603 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 0.18.2 > Environment: Solaris 10 x86 >Reporter: Jon Brisbin > Attachments: HADOOP-4603, id_instead_of_whoami.diff > > > A default installation as outlined in the docs won't start on Solaris 10 x86. > The "whoami" utility is in path "/usr/ucb" on Solaris 10, which isn't in the > standard PATH environment variable unless the user has added that > specifically. The documentation should reflect this. > Solaris 10 also seemed to throw NPEs if you didn't explicitly set the IP > address to bind the servers to. Simply overriding the IP address fixes the > problem. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4776) Windows installation fails with "bin/hadoop: line 243: c:\Program: command not found"
[
https://issues.apache.org/jira/browse/HADOOP-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-4776.
--
Resolution: Won't Fix
Closing this as won't fix now that Hadoop ships with batch files.
> Windows installation fails with "bin/hadoop: line 243: c:\Program: command
> not found"
> -
>
> Key: HADOOP-4776
> URL: https://issues.apache.org/jira/browse/HADOOP-4776
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Affects Versions: 0.19.0
> Environment: Windows
>Reporter: m costello
>Priority: Minor
> Original Estimate: 10m
> Remaining Estimate: 10m
>
> Perhaps a space in the path name is confusing Cygwin. The JAVA_HOME path is
> the default "C:\Program Files\Java\jdk1.6.0_11".
> Changing
> JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA}
> org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"`
> to
> JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} "${JAVA}"
> org.apache.hadoop.util.PlatformName | sed -e "s/ /_/g"`
> appear to correct the problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-4813) Avoid a buffer copy while replying to RPC requests.
[ https://issues.apache.org/jira/browse/HADOOP-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4813. -- Resolution: Incomplete Closing as stale given the RPC engine has been replaced. > Avoid a buffer copy while replying to RPC requests. > --- > > Key: HADOOP-4813 > URL: https://issues.apache.org/jira/browse/HADOOP-4813 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Raghu Angadi > Labels: newbie > > RPC server first serializes RPC response to a ByteArrayOutputStream and then > creates a new array to write to socket. For most responses the RPC handler is > able to write the entire response in-line. If we could use the same buffer > used by ByteArrayOutputStream, we can avoid this copy. > As mentioned in HADOOP-4802, yet another copy could be avoided (in most > cases) if we use a static direct buffer for the responses (not proposed for > this jira). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4815) S3FileSystem.renameRecursive(..) does not work correctly if src contains Java regular expression special characters
[
https://issues.apache.org/jira/browse/HADOOP-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-4815.
--
Resolution: Incomplete
S3 implementation was replaced. Closing as stale.
> S3FileSystem.renameRecursive(..) does not work correctly if src contains Java
> regular expression special characters
> ---
>
> Key: HADOOP-4815
> URL: https://issues.apache.org/jira/browse/HADOOP-4815
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>
> In S3FileSystem, the variable srcPath is not supposed to be a regular
> expression but is used as a regular expression in the line below.
> {code}
> Path newDst = new Path(oldSrcPath.replaceFirst(srcPath, dstPath));
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-4850) Fix IPC Client to not use UGI
[ https://issues.apache.org/jira/browse/HADOOP-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4850. -- Resolution: Incomplete I'm going to close this as stale, since it is probably too late. > Fix IPC Client to not use UGI > - > > Key: HADOOP-4850 > URL: https://issues.apache.org/jira/browse/HADOOP-4850 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Affects Versions: 0.20.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy > > Hadoop embraced JAAS via HADOOP-4348. > We need to fix IPC Client to use standard features of JAAS such as > LoginContext, Subject etc. rather than UserGroupInformation in the IPC > header, Client.Connection etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4853) Improvement to IPC
[ https://issues.apache.org/jira/browse/HADOOP-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4853. -- Resolution: Incomplete A lot of this has changed with the move to PB. Closing as stale. > Improvement to IPC > -- > > Key: HADOOP-4853 > URL: https://issues.apache.org/jira/browse/HADOOP-4853 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Affects Versions: 0.20.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy > > I'd like to propose an improvement for consideration given my experience of > working on HADOOP-4348: > Currently the first call doubles up as a 'connection setup' trigger. I'd like > to propose adding a new 'init' call which is always called by the clients for > connection setup. The advantages are: > * We could fold in the getProtocolVersion call into the setup call, this > ensures that the Server always checks for protocol versions, regardless of > whether the (malicious?) client does an explicit call for getProtocolVersion > or not. > * We could authorize the connection here. > * We could add to check to ensure that the Server instance actually > implements the protocol used by the client to communicate, rather than fail > on the first IPC call > The flip side being an extra round-trip. > Lets discuss. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4870) Ant test-patch goal (and test/bin/test-patch.sh script) fails without warnings if ANT_HOME environment variable is not set
[ https://issues.apache.org/jira/browse/HADOOP-4870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4870. -- Resolution: Won't Fix We've moved away from ant. Closing. > Ant test-patch goal (and test/bin/test-patch.sh script) fails without > warnings if ANT_HOME environment variable is not set > -- > > Key: HADOOP-4870 > URL: https://issues.apache.org/jira/browse/HADOOP-4870 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 0.18.2 > Environment: Ubuntu linux 8.10 >Reporter: Francesco Salbaroli >Priority: Minor > > A call to "ant test-patch" fails if ANT_HOME is not set as an environment > variable. > No errors if variable set with export or in /etc/environment -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4898) findbugs target and docs target which uses forrest is yet to be ported using IVY
[ https://issues.apache.org/jira/browse/HADOOP-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4898. -- Resolution: Won't Fix The forrest was cut down and the ants were eaten by birds. > findbugs target and docs target which uses forrest is yet to be ported using > IVY > - > > Key: HADOOP-4898 > URL: https://issues.apache.org/jira/browse/HADOOP-4898 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Giridharan Kesavan > > findbugs ant target is yet to be ported to use IVY for dependency management. > The reason being that , ivy can be used for resolving the findbugs.jar file > but the test-patch ant target uses the findbbugs bin directory. > docs ant target > docs uses forrest and java5, > forrest artifacts are unavailable on the centralized repo to be used through > ivy. > Thanks, > Giri -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4900) some dependencies are yet to be resolved using IVY
[ https://issues.apache.org/jira/browse/HADOOP-4900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4900. -- Resolution: Won't Fix Likely stale given all the rework on the build process. > some dependencies are yet to be resolved using IVY > --- > > Key: HADOOP-4900 > URL: https://issues.apache.org/jira/browse/HADOOP-4900 > Project: Hadoop Common > Issue Type: Wish > Components: build >Reporter: Giridharan Kesavan > > Though we are using ivy for resolving dependencies, not all the dependencies > are resolved through IVy. > The reason is the unavailability of the appropriate version of the artifacts > in the repository and the ambiguity of the version of the dependencies > At the moment beloe is the list of dependencies which are still resolved from > the local lib directories. > under the lib folder > commons-cli-2.0-SNAPSHOT.jar - yet to be available in the centralized repo > kfs-0.2.2.jar - not available in the > maven repo. > hsqldb-1.8.0.10.jar - latest available version is > under the lib/jsp-2.1 folder > jsp-2.1.jar - version # unknown > jsp-api-2.1.jar - version # unknown > under src/test/lib/ folder > ftplet-api-1.0.0-SNAPSHOT.jar - unavailable in > the maven repo > ftpserver-server-1.0.0-SNAPSHOT.jar - unavailable in the > maven repo > ftpserver-core-1.0.0-SNAPSHOT.jar - unavailable in the > maven repo > mina-core-2.0.0-M2-20080407.124109-12.jar- unavailable in the maven repo > under src/contrib/chukwa/lib > json.jar - version # > unknown > under src/contrib/thriftfs/lib > libthrift.jar - unavailable in the > maven repo. > Thanks, > Giri -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4917) fs -lsr does not align correctly when the username lengths are different
[
https://issues.apache.org/jira/browse/HADOOP-4917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-4917.
--
Resolution: Fixed
Closing this as fixed, as -R was added.
> fs -lsr does not align correctly when the username lengths are different
>
>
> Key: HADOOP-4917
> URL: https://issues.apache.org/jira/browse/HADOOP-4917
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
> Labels: newbie
>
> For example,
> {noformat}
> bash-3.2$ ./bin/hadoop fs -lsr /user
> drwx-- - nicholas supergroup 0 2008-12-18 15:17 /user/nicholas
> -rw-r--r-- 3 nn_sze supergroup 1366 2008-12-18 15:17
> /user/nicholas/a.txt
> drwx-- - tsz supergroup 0 2008-11-25 15:55 /user/tsz
> -rw--- 3 tsz supergroup 1366 2008-11-25 15:53 /user/tsz/r.txt
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-4953) config property mapred.child.java.opts has maximum length that generates NoClassDefFoundError if exceeded
[ https://issues.apache.org/jira/browse/HADOOP-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4953. -- Resolution: Won't Fix I'm closing this as won't fix since this parameter is mostly deprecated in lieu of map and reduce specific environment vars. > config property mapred.child.java.opts has maximum length that generates > NoClassDefFoundError if exceeded > - > > Key: HADOOP-4953 > URL: https://issues.apache.org/jira/browse/HADOOP-4953 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 0.19.0 > Environment: Amazon EC2 Ubuntu 8.04 hardy AMI" (Debian version > "lenny/sid") > JDK 1.6.0_07-b06 from Sun > kernel.ostype = Linux > kernel.osrelease = 2.6.21.7-2.fc8xen > kernel.version = #1 SMP Fri Feb 15 12:39:36 EST 2008 > powernow-k8: Found 1 Dual-Core AMD Opteron(tm) Processor 2218 HE processors > (version 2.00.00) >Reporter: Paul Baclace > > There is an unexpected max length for the value of config property > mapred.child.java.opts that, if exceeded, generates an opaque > NoClassDefFoundError in child tasks. > The max length for the value is 146 chars. A length of 147 chars will cause > the exception. For example, adding a single extra space between options will > convert a working jvm opts clause into one that always generates > NoClassDefFoundError when tasktrackers exec child tasks. > As laboriously diagnosed, conf/hadoop-site.xml was used to set the property > and runs were done on "Amazon EC2 Ubuntu 8.04 hardy AMI" (Debian version > "lenny/sid") using java 1.6.0_07-b06. Multiple slaves nodes were used and > after conf changes, stop-all.sh and start-all.sh were run before each test. > The job config props as found on the slave did not appear to have a truncated > or damaged value. It made no difference whether @taskid@ appeared at the end > or middle of the options and absence of @taskid@ did not eliminate the > problem. > This bug wastes considerable time because the error looks like a classpath > problem and even after the java opts property is suspected, a character > quoting or unsupported option seems more likely than a length limit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-4998) Implement a native OS runtime for Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-4998. -- Resolution: Fixed Closing as fixed as libhadoop.so has all sorts of random ... uhh.. stuff in it now. > Implement a native OS runtime for Hadoop > > > Key: HADOOP-4998 > URL: https://issues.apache.org/jira/browse/HADOOP-4998 > Project: Hadoop Common > Issue Type: New Feature > Components: native >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Attachments: hadoop-4998-1.patch > > > It would be useful to implement a JNI-based runtime for Hadoop to get access > to the native OS runtime. This would allow us to stop relying on exec'ing > bash to get access to information such as user-groups, process limits etc. > and for features such as chown/chgrp (org.apache.hadoop.util.Shell). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5059) 'whoami', 'topologyscript' calls failing with java.io.IOException: error=12, Cannot allocate memory
[
https://issues.apache.org/jira/browse/HADOOP-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-5059.
--
Resolution: Fixed
I'm going to close this as fixed.
A bunch of things have happened:
a) On certain platforms, java now uses posix_spawn() instead of fork().
b) Topology can now be provided by a class.
c) The whoami call has been removed.
So there are definitely ways to mitigate/eliminate this issue.
> 'whoami', 'topologyscript' calls failing with java.io.IOException: error=12,
> Cannot allocate memory
> ---
>
> Key: HADOOP-5059
> URL: https://issues.apache.org/jira/browse/HADOOP-5059
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Environment: On nodes with
> physical memory 32G
> Swap 16G
> Primary/Secondary Namenode using 25G of heap or more
>Reporter: Koji Noguchi
> Attachments: TestSysCall.java
>
>
> We've seen primary/secondary namenodes fail when calling whoami or
> topologyscripts.
> (Discussed as part of HADOOP-4998)
> Sample stack traces.
> Primary Namenode
> {noformat}
> 2009-01-12 03:57:27,381 WARN org.apache.hadoop.net.ScriptBasedMapping:
> java.io.IOException: Cannot run program
> "/path/topologyProgram" (in directory "/path"):
> java.io.IOException: error=12, Cannot allocate memory
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
> at org.apache.hadoop.util.Shell.run(Shell.java:134)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286)
> at
> org.apache.hadoop.net.ScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:122)
> at
> org.apache.hadoop.net.ScriptBasedMapping.resolve(ScriptBasedMapping.java:73)
> at
> org.apache.hadoop.dfs.FSNamesystem$ResolutionMonitor.run(FSNamesystem.java:1869)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
> allocate memory
> at java.lang.UNIXProcess.(UNIXProcess.java:148)
> at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
> ... 7 more
> 2009-01-12 03:57:27,381 ERROR org.apache.hadoop.fs.FSNamesystem: The resolve
> call returned null! Using /default-rack
> for some hosts
> 2009-01-12 03:57:27,381 INFO org.apache.hadoop.net.NetworkTopology: Adding a
> new node: /default-rack/55.5.55.55:50010
> {noformat}
> Secondary Namenode
> {noformat}
> 2008-10-09 02:00:58,288 ERROR org.apache.hadoop.dfs.NameNode.Secondary:
> java.io.IOException:
> javax.security.auth.login.LoginException: Login failed: Cannot run program
> "whoami": java.io.IOException:
> error=12, Cannot allocate memory
> at
> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250)
> at
> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275)
> at
> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257)
> at
> org.apache.hadoop.dfs.FSNamesystem.setConfigurationParameters(FSNamesystem.java:370)
> at org.apache.hadoop.dfs.FSNamesystem.(FSNamesystem.java:359)
> at
> org.apache.hadoop.dfs.SecondaryNameNode.doMerge(SecondaryNameNode.java:340)
> at
> org.apache.hadoop.dfs.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:312)
> at
> org.apache.hadoop.dfs.SecondaryNameNode.run(SecondaryNameNode.java:223)
> at java.lang.Thread.run(Thread.java:619)
> at
> org.apache.hadoop.dfs.FSNamesystem.setConfigurationParameters(FSNamesystem.java:372)
> at org.apache.hadoop.dfs.FSNamesystem.(FSNamesystem.java:359)
> at
> org.apache.hadoop.dfs.SecondaryNameNode.doMerge(SecondaryNameNode.java:340)
> at
> org.apache.hadoop.dfs.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:312)
> at
> org.apache.hadoop.dfs.SecondaryNameNode.run(SecondaryNameNode.java:223)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-5061) Update chinese documentation for default configuration
[ https://issues.apache.org/jira/browse/HADOOP-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5061. -- Resolution: Incomplete Closing this as stale. > Update chinese documentation for default configuration > -- > > Key: HADOOP-5061 > URL: https://issues.apache.org/jira/browse/HADOOP-5061 > Project: Hadoop Common > Issue Type: Improvement > Components: documentation >Reporter: Sharad Agarwal > > The chinese documentation needs to be updated as per HADOOP-4828 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5069) add a Hadoop-centric junit test result listener
[ https://issues.apache.org/jira/browse/HADOOP-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5069. -- Resolution: Fixed We replaced ant with maven. Closing as fixed. > add a Hadoop-centric junit test result listener > --- > > Key: HADOOP-5069 > URL: https://issues.apache.org/jira/browse/HADOOP-5069 > Project: Hadoop Common > Issue Type: New Feature > Components: test >Affects Versions: 0.21.0 >Reporter: Steve Loughran >Priority: Minor > > People are encountering different problems with hadoop's unit tests, defects > currently being WONTFIX'd > # HADOOP-5001 : Junit tests that time out don't write any test progress > related logs > # HADOOP-4721 : OOM in .TestSetupAndCleanupFailure > There is a root cause here, the XmlResultFormatter of Ant buffers everything > before writing out a DOM. Too much logged: OOM and no output. Timeout: kill > and no output. > We could add a new logger class to hadoop and then push it back into Ant once > we were happy, or keep it separate if we had specific dependencies (like on > hadoop-dfs API) that they lacked. > Some ideas > # stream XML to disk. We would have to put the test summary at the end; could > use XSL to generate HTML and the classic XML content > # stream XHTML to disk. Makes it readable as you go along; makes the XSL work > afterwards harder. > # push out results as records to a DFS. There's a problem here in that this > needs to be a different DFS from that you are testing, yet it needs to be > compatible with the client. > Item #3 would be interesting but doing it inside JUnit is too dangerous > classpath and config wise. Better to have Ant do the copy afterwards. What is > needed then is a way to easily append different tests to the same DFS file in > a way that tools can analyse them all afterwards. The copy is easy -add a new > Ant resource for that- but the choice of format is trickier. > Here's some work I did on this a couple of years back; I've not done much > since then: > http://people.apache.org/~stevel/slides/distributed_testing_with_smartfrog_slides.pdf > Is anyone else interested in exploring this? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5102) Split build script for building core, hdfs and mapred separately
[ https://issues.apache.org/jira/browse/HADOOP-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5102. -- Resolution: Fixed > Split build script for building core, hdfs and mapred separately > > > Key: HADOOP-5102 > URL: https://issues.apache.org/jira/browse/HADOOP-5102 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Reporter: Sharad Agarwal > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5109) TestKillCompletedJob is failing intermittetnly when run as part of test-core
[
https://issues.apache.org/jira/browse/HADOOP-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-5109.
--
Resolution: Fixed
> TestKillCompletedJob is failing intermittetnly when run as part of test-core
>
>
> Key: HADOOP-5109
> URL: https://issues.apache.org/jira/browse/HADOOP-5109
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
>Reporter: Jakob Homan
> Attachments: TEST-org.apache.hadoop.mapred.TestKillCompletedJob.txt
>
>
> TestKillCompletedJob fails most times when run as part of test-core, but
> succeeds when run by itself.
> {noformat}
> Testcase: testKillCompJob took 8.048 sec
> Caused an ERROR
> Job failed!
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1343)
> at
> org.apache.hadoop.mapred.TestKillCompletedJob.launchWordCount(TestKillCompletedJob.java:78)
> at
> org.apache.hadoop.mapred.TestKillCompletedJob.testKillCompJob(TestKillCompletedJob.java:112)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-5112) Upgrade Clover to 2.4.2 and enable Test Optimization in HADOOP
[ https://issues.apache.org/jira/browse/HADOOP-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5112. -- Resolution: Incomplete Closing as stale. > Upgrade Clover to 2.4.2 and enable Test Optimization in HADOOP > -- > > Key: HADOOP-5112 > URL: https://issues.apache.org/jira/browse/HADOOP-5112 > Project: Hadoop Common > Issue Type: Task > Components: build >Reporter: Nick Pellow > > The current [Hadoop > build|http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-trunk/clover/] > on Hudson is using Clover 1.3.13. > I will attach a patch to the build.xml and the clover 2.4.2 jar to this issue. > Test Optimization works by only running tests for which code has changed. > This can be used both in a CI environment and on a developers machine to make > running tests faster. > Clover will also run tests which previously failed. Modified source code is > detected at compile time and then used to select the tests that get run. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5116) TestSocketIOWithTimeout fails under AIX - TIMEOUT error.
[ https://issues.apache.org/jira/browse/HADOOP-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5116. -- Resolution: Incomplete Closing as stale. > TestSocketIOWithTimeout fails under AIX - TIMEOUT error. > - > > Key: HADOOP-5116 > URL: https://issues.apache.org/jira/browse/HADOOP-5116 > Project: Hadoop Common > Issue Type: Test > Components: test >Affects Versions: 0.18.2 > Environment: AIX >Reporter: Bill Habermaas >Priority: Minor > Attachments: javacore.20090126.144729.376858.0001.txt > > > This test expects an exception to occur when read/writing a closed socket. > Under AIX this does not occur and results in a loop. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5115) TestLocalDirAllocator fails under AIX
[ https://issues.apache.org/jira/browse/HADOOP-5115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5115. -- Resolution: Incomplete Closing as stale. > TestLocalDirAllocator fails under AIX > -- > > Key: HADOOP-5115 > URL: https://issues.apache.org/jira/browse/HADOOP-5115 > Project: Hadoop Common > Issue Type: Test > Components: test >Affects Versions: 0.18.2 > Environment: AIX >Reporter: Bill Habermaas >Priority: Minor > > TestLocalDirAllocator fails when running under AIX for the same reasons as > CYGWIN under Windows (as noted in the test source code comments). AIX allows > the writing of a file in a directory that is marked read-only. This breaks > the test. If the test is changed to sense for AIX (as it does for windows) > then the usefulness of this unit test is questionable other than exposing an > interesting anomoly in the native file system. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5179) FileSystem#copyToLocalFile shouldn't copy .crc files
[ https://issues.apache.org/jira/browse/HADOOP-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5179. -- Resolution: Duplicate Closing as a dupe then. > FileSystem#copyToLocalFile shouldn't copy .crc files > > > Key: HADOOP-5179 > URL: https://issues.apache.org/jira/browse/HADOOP-5179 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.19.0 >Reporter: Nathan Marz > > The .crc files shouldn't be copied locally, as they are an internal Hadoop > filesystem thing. This is causing the following problem for me: > I sometimes copy a directory from HDFS locally, modify those files, and then > reupload them somewhere else in HDFS. I then get checksum errors on the > re-upload. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5195) Unit tests for TestProxyUgiManager and TestHdfsProxy consistently failing on trunk builds
[ https://issues.apache.org/jira/browse/HADOOP-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5195. -- Resolution: Fixed Closing as stale. > Unit tests for TestProxyUgiManager and TestHdfsProxy consistently failing on > trunk builds > - > > Key: HADOOP-5195 > URL: https://issues.apache.org/jira/browse/HADOOP-5195 > Project: Hadoop Common > Issue Type: Bug > Components: test >Affects Versions: 0.21.0 >Reporter: Lee Tucker >Priority: Minor > > of the last 10 trunk builds, these unit tests have failed in about 50% of the > builds. Trunk builds have been failing unit tests consistently for as far > back as I can see in hudson. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5238) ivy publish and ivy integration does not cleanly work
[
https://issues.apache.org/jira/browse/HADOOP-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-5238.
--
Resolution: Fixed
> ivy publish and ivy integration does not cleanly work
> -
>
> Key: HADOOP-5238
> URL: https://issues.apache.org/jira/browse/HADOOP-5238
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 0.19.0
>Reporter: Stefan Groschupf
>Assignee: Giridharan Kesavan
>Priority: Minor
>
> As far I understand the goal using ivy for hadoop is to be able intgrating
> hadoop easily in thirdparty builds that uses transient dependency tools like
> ivy or maven.
> The way ivy is currently integrated a couple hick ups.
> + the generated artifact files have names that can't be used for maven or
> ivy. e.g. hadoop-version-core but standard would be hadoop-core-version. This
> effects all those generated artifact files like hadoop-version-example etc.
> This is caused by the use of ${final.name}-core.jar
> + This conflicts with the use of "${version}" in ivy.xml organisation="org.apache.hadoop" module="${ant.project.name}"
> revision="${version}">. The result will be a error report by ivy that found
> artifact and defined artifact name are different.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-5261) HostsFileReader does not properly implement concurrency support
[ https://issues.apache.org/jira/browse/HADOOP-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5261. -- Resolution: Fixed Fixed. > HostsFileReader does not properly implement concurrency support > --- > > Key: HADOOP-5261 > URL: https://issues.apache.org/jira/browse/HADOOP-5261 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jakob Homan >Assignee: Jakob Homan > Attachments: HADOOP-5261.patch > > > As currently implemented, the class HostsFileReader does not properly allow > concurrent access. > It maintains two Sets and manipulates them within synchronized fields, but > provides accessor methods that publish unsynchronized access to the sets' > references (getHosts() and getExcludedHosts()). The sets are implemented as > HashSets, which are not thread safe. This can allow a method to obtain a > reference to a set that may be modified concurrently by the HostsFileReader. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5295) JobTracker can hold the list of lost TaskTrackers instead of removing them completely.
[ https://issues.apache.org/jira/browse/HADOOP-5295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5295. -- Resolution: Won't Fix Won't Fix due to YARN. > JobTracker can hold the list of lost TaskTrackers instead of removing them > completely. > -- > > Key: HADOOP-5295 > URL: https://issues.apache.org/jira/browse/HADOOP-5295 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Vinod Kumar Vavilapalli > > Having the name and possibly the time for which it has been lost and > displaying the info via client UI as well as web UI will help in recognizing > problematic nodes easily and quickly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5312) Not all core javadoc are checked by Hudson
[ https://issues.apache.org/jira/browse/HADOOP-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5312. -- Resolution: Fixed With Mavenization, etc, I think this is fixed if not just stale now. > Not all core javadoc are checked by Hudson > -- > > Key: HADOOP-5312 > URL: https://issues.apache.org/jira/browse/HADOOP-5312 > Project: Hadoop Common > Issue Type: Task > Components: build >Reporter: Tsz Wo Nicholas Sze > > Since "ant javadoc" does not generate all core javadocs, some javadocs (e.g. > HDFS javadocs) are not checked by Hudson. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5345) JobID is deprecated but there are no references to classes that are replacing it.
[ https://issues.apache.org/jira/browse/HADOOP-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5345. -- Resolution: Won't Fix I suppose at this point, this is just stale given the deprecration -> undeprecation that happened. > JobID is deprecated but there are no references to classes that are replacing > it. > - > > Key: HADOOP-5345 > URL: https://issues.apache.org/jira/browse/HADOOP-5345 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 0.20.0 >Reporter: Santhosh Srinivasan > > JobID is deprecated but there are no references to classes that are replacing > it. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5388) Add ids to tables in JSP pages to ease scraping of the data.
[ https://issues.apache.org/jira/browse/HADOOP-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5388. -- Resolution: Incomplete I'm going to close this as stale. First, the UIs have changed tremendously. Secondly, all of this data is now available via JSON, JMX, and various shell utilities. > Add ids to tables in JSP pages to ease scraping of the data. > > > Key: HADOOP-5388 > URL: https://issues.apache.org/jira/browse/HADOOP-5388 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Sreekanth Ramakrishnan >Priority: Minor > Attachments: HADOOP-5388.patch > > > Currently, the tables which are generated by the JSP pages are lacking id's. > If the tables had id then it would ease the pain of writing scrapping > utilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5402) TaskTracker ignores most RemoteExceptions from heartbeat processing
[
https://issues.apache.org/jira/browse/HADOOP-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-5402.
--
Resolution: Incomplete
This is probably stale.
> TaskTracker ignores most RemoteExceptions from heartbeat processing
> ---
>
> Key: HADOOP-5402
> URL: https://issues.apache.org/jira/browse/HADOOP-5402
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 0.21.0
>Reporter: Owen O'Malley
>
> The code in TaskTracker.offerService looks like:
> {code}
> } catch (RemoteException re) {
> String reClass = re.getClassName();
> if (DisallowedTaskTrackerException.class.getName().equals(reClass)) {
> LOG.info("Tasktracker disallowed by JobTracker.");
> return State.DENIED;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-5436) job history directory grows without bound, locks up job tracker on new job submission
[ https://issues.apache.org/jira/browse/HADOOP-5436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5436. -- Resolution: Fixed > job history directory grows without bound, locks up job tracker on new job > submission > - > > Key: HADOOP-5436 > URL: https://issues.apache.org/jira/browse/HADOOP-5436 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.19.0, 0.20.0, 0.20.1, 0.20.2 >Reporter: Tim Williamson > Attachments: HADOOP-5436.patch > > > An unpleasant surprise upgrading to 0.19: requests to jobtracker.jsp would > take a long time or even time out whenever new jobs where submitted. > Investigation showed the call to JobInProgress.initTasks() was calling > JobHistory.JobInfo.logSubmitted() which in turn was calling > JobHistory.getJobHistoryFileName() which was pegging the CPU for a couple > minutes. Further investigation showed the were 200,000+ files in the job > history folder -- and every submission was creating a FileStatus for them > all, then applying a regular expression to just the name. All this just on > the off chance the job tracker had been restarted (see HADOOP-3245). To make > matters worse, these files cannot be safely deleted while the job tracker is > running, as the disappearance of a history file at the wrong time causes a > FileNotFoundException. > So to summarize the issues: > - having Hadoop default to storing all the history files in a single > directory is a Bad Idea > - doing expensive processing of every history file on every job submission is > a Worse Idea > - doing expensive processing of every history file on every job submission > while holding a lock on the JobInProgress object and thereby blocking the > jobtracker.jsp from rendering is a Terrible Idea (note: haven't confirmed > this, but a cursory glance suggests that's what's going on) > - not being able to clean up the mess without taking down the job tracker is > just Unfortunate -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5439) FileSystem.create() with overwrite param specified sometimes takes a long time to return.
[
https://issues.apache.org/jira/browse/HADOOP-5439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer resolved HADOOP-5439.
--
Resolution: Fixed
> FileSystem.create() with overwrite param specified sometimes takes a long
> time to return.
> -
>
> Key: HADOOP-5439
> URL: https://issues.apache.org/jira/browse/HADOOP-5439
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 0.19.0
>Reporter: He Yongqiang
>
> If a file already exists, it takes a long time for the overwrite create to
> return.
> {code}
> fs.create(path_1, true);
> {code}
> sometimes takes a long time.
> Instead, the code:
> {code}
> if (fs.exists(path_1))
> fs.delete(path_1);
> fs.create(path_1);
> {code}
> works pretty well.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HADOOP-5566) JobTrackerMetricsInst should probably keep track of JobFailed
[ https://issues.apache.org/jira/browse/HADOOP-5566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5566. -- Resolution: Incomplete I'm just going to close this as stale. > JobTrackerMetricsInst should probably keep track of JobFailed > - > > Key: HADOOP-5566 > URL: https://issues.apache.org/jira/browse/HADOOP-5566 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jerome Boulon > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5602) existing Bzip2Codec supported in hadoop 0.19/0.20 skipps the input records when input bzip2 compressed files is made up of concatinating multiple .bz2 files.
[ https://issues.apache.org/jira/browse/HADOOP-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5602. -- Resolution: Fixed > existing Bzip2Codec supported in hadoop 0.19/0.20 skipps the input records > when input bzip2 compressed files is made up of concatinating multiple .bz2 > files. > -- > > Key: HADOOP-5602 > URL: https://issues.apache.org/jira/browse/HADOOP-5602 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 0.19.0, 0.19.1 >Reporter: Suhas Gogate > > Until the Bzip2Codec supports concatenated compressed bzip files as input it > should detect it and throw an error to indicate input is not compatible... > (see the related JIRA https://issues.apache.org/jira/browse/HADOOP-5601) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5617) make chukwa log4j configuration more transparent from hadoop
[ https://issues.apache.org/jira/browse/HADOOP-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5617. -- Resolution: Duplicate HADOOP-9902 fixes the env var issues. > make chukwa log4j configuration more transparent from hadoop > > > Key: HADOOP-5617 > URL: https://issues.apache.org/jira/browse/HADOOP-5617 > Project: Hadoop Common > Issue Type: Improvement > Components: build > Environment: Redhat 5.1, Java 6 >Reporter: Eric Yang >Priority: Minor > > The current log4j appender retro fitting to hadoop is less than ideal. In > theory, the log4j appender configuration should be changable by the > environment scripts. This ticket is to track any changes required to make > hadoop log4j configuration more portable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5682) ant clean does not clean the generated api docs
[ https://issues.apache.org/jira/browse/HADOOP-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5682. -- Resolution: Fixed This is almost certainly stale by now. > ant clean does not clean the generated api docs > --- > > Key: HADOOP-5682 > URL: https://issues.apache.org/jira/browse/HADOOP-5682 > Project: Hadoop Common > Issue Type: Bug > Components: build >Reporter: Giridharan Kesavan >Assignee: Giridharan Kesavan > Labels: newbie > Attachments: hadoop-5682.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10878) Hadoop servlets need ACLs
Allen Wittenauer created HADOOP-10878: - Summary: Hadoop servlets need ACLs Key: HADOOP-10878 URL: https://issues.apache.org/jira/browse/HADOOP-10878 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Allen Wittenauer As far as I'm aware, once a user gets past the HTTP-level authentication, all servlets available on that port are available to the user. This is a security hole as there is some information and services that we don't want every user to be able to access or only want them to access from certain locations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5722) HTTP metrics interface enable/disable must be configurable
[ https://issues.apache.org/jira/browse/HADOOP-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5722. -- Resolution: Fixed I'm going to dupe this to a new bug, as it isn't just metrics that has this problem now. > HTTP metrics interface enable/disable must be configurable > -- > > Key: HADOOP-5722 > URL: https://issues.apache.org/jira/browse/HADOOP-5722 > Project: Hadoop Common > Issue Type: Bug > Components: metrics, security >Reporter: Marco Nicosia > > HADOOP-5469 added a convenient end-run around JMX authentication by revealing > the same metrics over HTTP. That's cool, but we need to secure all accesses > to our Hadoop cluster, so while this may be enabled by default, we need some > configurable way to disable the unauthenticated port. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5760) Task process hanging on an RPC call
[ https://issues.apache.org/jira/browse/HADOOP-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5760. -- Resolution: Duplicate > Task process hanging on an RPC call > --- > > Key: HADOOP-5760 > URL: https://issues.apache.org/jira/browse/HADOOP-5760 > Project: Hadoop Common > Issue Type: Bug > Components: ipc >Reporter: Devaraj Das > > On a random node on a cluster, I found one task process waiting on an RPC > call. The process has been in that state for a few days at least. > "main" prio=10 tid=0x08069400 nid=0x6f52 in Object.wait() > [0xf7e6c000..0xf7e6d1f8] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0xf1215700> (a org.apache.hadoop.ipc.Client$Call) > at java.lang.Object.wait(Object.java:485) > at org.apache.hadoop.ipc.Client.call(Client.java:725) > - locked <0xf1215700> (a org.apache.hadoop.ipc.Client$Call) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > at org.apache.hadoop.mapred.$Proxy0.statusUpdate(Unknown Source) > at org.apache.hadoop.mapred.Task.statusUpdate(Task.java:691) > at org.apache.hadoop.mapred.Task.taskCleanup(Task.java:795) > at org.apache.hadoop.mapred.Child.main(Child.java:176) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5791) to downgrade commons-cli from 2.0 to 1.2
[ https://issues.apache.org/jira/browse/HADOOP-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5791. -- Resolution: Won't Fix Long stale. > to downgrade commons-cli from 2.0 to 1.2 > - > > Key: HADOOP-5791 > URL: https://issues.apache.org/jira/browse/HADOOP-5791 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giridharan Kesavan >Assignee: Devaraj Das > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-6766) Spill can fail with bad call to Random
[ https://issues.apache.org/jira/browse/HADOOP-6766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-6766. -- Resolution: Cannot Reproduce > Spill can fail with bad call to Random > -- > > Key: HADOOP-6766 > URL: https://issues.apache.org/jira/browse/HADOOP-6766 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 0.20.2 >Reporter: Peter Arthur Ciccolo >Priority: Minor > > java.lang.IllegalArgumentException: n must be positive > at java.util.Random.nextInt(Random.java:250) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:243) > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:289) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124) > at > org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1221) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1129) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:159) > confChanged assumes that the list of dirs it creates > (LocalDirAllocator.java:215) has at least one element in it by the end of the > function. If, for each local dir, either the conditional on line 221 is > false, or the call to DiskChecker.checkDir() throws an exception, this > assumption will not hold. In this case, dirIndexRandomizer.nextInt() is > called on the number of elements in dirs, which is 0. Since > dirIndexRandomizer (195) is an instance of Random(), it needs a positive > (non-zero) argument to nextInt(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5772) Implement a 'refreshable' configuration system with right access-controls etc.
[ https://issues.apache.org/jira/browse/HADOOP-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5772. -- Resolution: Duplicate > Implement a 'refreshable' configuration system with right access-controls etc. > -- > > Key: HADOOP-5772 > URL: https://issues.apache.org/jira/browse/HADOOP-5772 > Project: Hadoop Common > Issue Type: New Feature > Components: conf >Reporter: Arun C Murthy > > We have various bits and pieces of code to refresh certain configuration > files, various components to restrict access to who can actually refresh the > configs etc. > I propose we start thinking about a simple system to support this as a > first-class citizen in Hadoop Core... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10879) Rename *-env.sh in the tree to *-env.sh.example
Allen Wittenauer created HADOOP-10879: - Summary: Rename *-env.sh in the tree to *-env.sh.example Key: HADOOP-10879 URL: https://issues.apache.org/jira/browse/HADOOP-10879 Project: Hadoop Common Issue Type: Improvement Reporter: Allen Wittenauer With HADOOP-9902 in place, we don't have to ship *-env.sh called as such and only provide examples. This goes a long way with being able to upgrade the binaries in place since we would no longer overwrite those files upon extraction. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5888) Improve test run time by avoiding 0.0.0.0 lookups
[ https://issues.apache.org/jira/browse/HADOOP-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5888. -- Resolution: Fixed I'm going to close this as half-fixed/half-won't fix due to some tests needing to be on 0.0.0.0 to make sure that we test that configuration since we give it to users. > Improve test run time by avoiding 0.0.0.0 lookups > - > > Key: HADOOP-5888 > URL: https://issues.apache.org/jira/browse/HADOOP-5888 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Todd Lipcon > Attachments: hadoop-5888.txt > > > HADOOP-3694 discusses the fact that 0.0.0.0 is slower to reverse than > 127.0.0.1 on a lot of systems. The replacing of 0.0.0.0 with 127.0.0.1 was > only partially completed in that ticket. This ticket is to track continued > work on that front. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HADOOP-5889) Allow writing to output directories that exist, as long as they are empty
[ https://issues.apache.org/jira/browse/HADOOP-5889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-5889. -- Resolution: Incomplete I'm just going to close this as stale, esp given s3 allegedly works. > Allow writing to output directories that exist, as long as they are empty > - > > Key: HADOOP-5889 > URL: https://issues.apache.org/jira/browse/HADOOP-5889 > Project: Hadoop Common > Issue Type: Improvement > Components: fs >Affects Versions: 0.18.3 >Reporter: Ian Nowland > Attachments: HADOOP-5889-0.patch > > > The current behavior in FileOutputFormat.checkOutputSpecs is to fail if the > path specified by mapred.output.dir exists at the start of the job. This is > to protect from accidentally overwriting existing data. There seems no harm > then in slightly relaxing this check to allow the case for the output to > exist if it is an empty directory. > At a minimum this would allow outputting to the root of S3N buckets, which is > currently impossible (https://issues.apache.org/jira/browse/HADOOP-5805). -- This message was sent by Atlassian JIRA (v6.2#6252)
