[jira] [Created] (HADOOP-9324) Out of date API document

[Hao Zhong (JIRA)]

Hao Zhong created HADOOP-9324:
-

 Summary: Out of date API document
 Key: HADOOP-9324
 URL: https://issues.apache.org/jira/browse/HADOOP-9324
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Hao Zhong


The documentation is out of date. Some code references are broken:
1. 
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FSDataInputStream.html
"All Implemented Interfaces:
Closeable, DataInput, *org.apache.hadoop.fs.ByteBufferReadable*, 
*org.apache.hadoop.fs.HasFileDescriptor*, PositionedReadable, Seekable "

2.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Cluster.html
renewDelegationToken(*org.apache.hadoop.security.token.Token*
 token)
  Deprecated. Use Token.renew(*org.apache.hadoop.conf.Configuration*) 
instead

3.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/JobConf.html
"Use MRAsyncDiskService.moveAndDeleteAllVolumes instead. "
I cannot find the MRAsyncDiskService class in the documentation of 2.0.3. 

4.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/join/CompositeRecordReader.html
 "protected *org.apache.hadoop.mapred.join.CompositeRecordReader.JoinCollector* 
jc"
Please globally search JoinCollector. It is deleted, but mentioned many times 
in the current documentation.

5.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/OutputCommitter.html
"abortJob(JobContext context, *org.apache.hadoop.mapreduce.JobStatus.State 
runState*)"  
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html
"public *org.apache.hadoop.mapreduce.JobStatus.State* getJobState()"

4.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/SequenceFileOutputFormat.html
" static *org.apache.hadoop.io.SequenceFile.CompressionType* 
getOutputCompressionType"
" static *org.apache.hadoop.io.SequenceFile.Reader[]*   getReaders"

5.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/TaskCompletionEvent.html
"Returns enum Status.SUCESS or Status.FAILURE."->Status.SUCCEEDED? 

6.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html
" static *org.apache.hadoop.mapreduce.Job.TaskStatusFilter* 
getTaskOutputFilter"
"  org.apache.hadoop.mapreduce.TaskReport[] getTaskReports(TaskType type) "

7.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Reducer.html
"cleanup(*org.apache.hadoop.mapreduce.Reducer.Context* context) "

8.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/SequenceFileOutputFormat.html
 "static *org.apache.hadoop.io.SequenceFile.CompressionType*
getOutputCompressionType(JobConf conf)
  Get the *SequenceFile.CompressionType* for the output SequenceFile."
" static *org.apache.hadoop.io.SequenceFile.Reader[]*   getReaders" 

9.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/partition/InputSampler.html
"writePartitionFile(Job job, 
*org.apache.hadoop.mapreduce.lib.partition.InputSampler.Sampler* sampler) "

10.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/partition/TotalOrderPartitioner.html
contain JobContextImpl.getNumReduceTasks() - 1 keys. 
The JobContextImpl class is already deleted.

11. 
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/OutputCommitter.html
"Note that this is invoked for jobs with final runstate as 
JobStatus.State.FAILED or JobStatus.State.KILLED."->JobStatus.FAILED 
JobStatus.KILLED?

12.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/TaskAttemptContext.html
"All Superinterfaces:
JobContext, *org.apache.hadoop.mapreduce.MRJobConfig*, Progressable, 
TaskAttemptContext "

13.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/file/FileContext.html
"All Implemented Interfaces:
*org.apache.hadoop.metrics.MetricsContext*"

14.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/spi/AbstractMetricsContext.html
   
"*org.apache.hadoop.metrics.MetricsRecord*  createRecord(String recordName)"

15. 
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/net/DNSToSwitchMapping.html
"If a name cannot be resolved to a rack, the implementation should return 
NetworkTopology.DEFAULT_RACK."
NetworkTopology is deleted.

16.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html
"myprefix.sink.file.class=org.hadoop.metrics2.sink.FileSink" ->
org.apache.hadoop.metrics2.sink.FileSink?
"org.apache.hadoop.metrics2.impl" -> The package is not found.

17.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/ha/HAServiceTarget.html
" abstract  *org.apache.hadoop.ha.NodeFencer*   getFencer() "

18.http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/MarkableIterator.html
"MarkableIterator is a wrapper iterator class that implements the 
MarkableIte

[jira] [Created] (HADOOP-9323) Typos in API documentation

[Hao Zhong (JIRA)]

Hao Zhong created HADOOP-9323:
-

 Summary: Typos in API documentation
 Key: HADOOP-9323
 URL: https://issues.apache.org/jira/browse/HADOOP-9323
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Hao Zhong
Priority: Critical


Some typos are as follows:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/ChecksumFileSystem.html
basice->basic

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileContext.html
sytem->system

http://hadoop.apache.org/docs/current/api/index.html?org/apache/hadoop/fs/RawLocalFileSystem.html
http://hadoop.apache.org/docs/current/api/index.html?org/apache/hadoop/fs/FilterFileSystem.html
inital->initial

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/TrashPolicy.html
paramater->parameter

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/PositionedReadable.html
equalt->equal

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/BytesWritable.html
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/record/Buffer.html
seqeunce->sequence

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/Text.html
instatiation->instantiation

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/record/RecordOutput.html
alll->all

Please revise the documentation. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-9112) test-patch should -1 for @Tests without a timeout

[Surenkumar Nihalani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Surenkumar Nihalani resolved HADOOP-9112.
-

Resolution: Fixed

> test-patch should -1 for @Tests without a timeout
> -
>
> Key: HADOOP-9112
> URL: https://issues.apache.org/jira/browse/HADOOP-9112
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Todd Lipcon
>Assignee: Surenkumar Nihalani
> Fix For: 3.0.0
>
> Attachments: HADOOP-9112-1.patch, HADOOP-9112-2.patch, 
> HADOOP-9112-3.patch, HADOOP-9112-4.patch, HADOOP-9112-5.patch, 
> HADOOP-9112-6.patch, HADOOP-9112-7.patch
>
>
> With our current test running infrastructure, if a test with no timeout set 
> runs too long, it triggers a surefire-wide timeout, which for some reason 
> doesn't show up as a failed test in the test-patch output. Given that, we 
> should require that all tests have a timeout set, and have test-patch enforce 
> this with a simple check

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-9309) test failures on Windows due to UnsatisfiedLinkError in NativeCodeLoader#buildSupportsSnappy

[Suresh Srinivas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HADOOP-9309.
-

   Resolution: Fixed
Fix Version/s: trunk-win
 Hadoop Flags: Reviewed

I committed the patch to branch-trunk-win. Thank you Aprit!

Thank you Chris for the review.

> test failures on Windows due to UnsatisfiedLinkError in 
> NativeCodeLoader#buildSupportsSnappy
> 
>
> Key: HADOOP-9309
> URL: https://issues.apache.org/jira/browse/HADOOP-9309
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: native, util
>Affects Versions: trunk-win
>Reporter: Chris Nauroth
>Assignee: Arpit Agarwal
> Fix For: trunk-win
>
> Attachments: HADOOP-9309.1.patch, HADOOP-9309.patch
>
>
> Checking for Snappy support calls native method 
> {{NativeCodeLoader#buildSupportsSnappy}}.  This method has not been 
> implemented for Windows in hadoop.dll, so it throws {{UnsatisfiedLinkError}}.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9322) LdapGroupsMapping doesn't seem to set a timeout for its directory search

[Harsh J (JIRA)]

Harsh J created HADOOP-9322:
---

 Summary: LdapGroupsMapping doesn't seem to set a timeout for its 
directory search
 Key: HADOOP-9322
 URL: https://issues.apache.org/jira/browse/HADOOP-9322
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 2.0.3-alpha
Reporter: Harsh J
Priority: Minor


We don't appear to be setting a timeout via 
http://docs.oracle.com/javase/6/docs/api/javax/naming/directory/SearchControls.html#setTimeLimit(int)
 before we search with 
http://docs.oracle.com/javase/6/docs/api/javax/naming/directory/DirContext.html#search(javax.naming.Name,%20java.lang.String,%20javax.naming.directory.SearchControls).

This may occasionally lead to some unwanted NN pauses due to lock-holding on 
the operations that do group lookups. A timeout is better to define than rely 
on "0" (infinite wait).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: timeout is now requested to be on all tests

[Surenkumar Nihalani]

Just wanted to give everyone a heads up. Due to this change, I have introduced 
a bug. 

It's returning opposite value of what it's supposed to. 
So, +1 of all tests you change brings up a -1 overall and vice versa. 

I have reopened HADOOP-9112 and uploaded the patch to reverse to reverse to the 
correct return values. Once a committer goes ahead with, it will be corrected.

Sorry about the inconvenience. 

Thanks,
Surenkumar Nihalani.

On Feb 21, 2013, at 5:28 AM, Steve Loughran  wrote:

> thanks, just seen and commented on this.
> 
> IF we're going to have test timeouts, we need a good recommended default
> value for all tests except the extra slow ones.
> 
> Or
> 1. we just use our own JUnit fork that sets up a better default value than
> 0. I don't know how Ken would react to that.
> 2. we get maven sure to support a programmable (default) timeout for every
> test case. This is probably easier, and would work for all existing tests
> too.
> 
> Has anyone asked the Maven team about options here?
> 
> On 20 February 2013 16:46, Robert Evans  wrote:
> 
>> Sorry about cross posting, but this will impact all developers and I
>> wanted to give you all a heads-up.
>> 
>> HADOOP-9112 was just
>> checked it.  This means that the pre commit build will now give a –1 for
>> any patch with junit tests that do not include a timeout option.  See
>> http://junit.sourceforge.net/javadoc/org/junit/Test.html for more info on
>> that.  This is to avoid surefire timing out junit when it gets stuck and
>> not giving any real feedback on which test failed.
>> 
>> --Bobby
>> 

[jira] [Resolved] (HADOOP-9313) Remove spurious mkdir from hadoop-config.cmd

[Suresh Srinivas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Suresh Srinivas resolved HADOOP-9313.
-

   Resolution: Fixed
Fix Version/s: trunk-win
 Hadoop Flags: Reviewed

+1 for the patch. I committed it to branch-trunk-win.

Thank you Ivan!

> Remove spurious mkdir from hadoop-config.cmd
> 
>
> Key: HADOOP-9313
> URL: https://issues.apache.org/jira/browse/HADOOP-9313
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: scripts
>Affects Versions: trunk-win
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
> Fix For: trunk-win
>
> Attachments: HADOOP-9313.branch-trunk-win.cmd.patch
>
>
> The following mkdir seems to have been accidentally added to Windows cmd 
> script and should be removed:
> {code}
> mkdir c:\tmp\dir1
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: timeout is now requested to be on all tests

[Steve Loughran]

thanks, just seen and commented on this.

IF we're going to have test timeouts, we need a good recommended default
value for all tests except the extra slow ones.

Or
1. we just use our own JUnit fork that sets up a better default value than
0. I don't know how Ken would react to that.
2. we get maven sure to support a programmable (default) timeout for every
test case. This is probably easier, and would work for all existing tests
too.

Has anyone asked the Maven team about options here?

On 20 February 2013 16:46, Robert Evans  wrote:

> Sorry about cross posting, but this will impact all developers and I
> wanted to give you all a heads-up.
>
> HADOOP-9112 was just
> checked it.  This means that the pre commit build will now give a –1 for
> any patch with junit tests that do not include a timeout option.  See
> http://junit.sourceforge.net/javadoc/org/junit/Test.html for more info on
> that.  This is to avoid surefire timing out junit when it gets stuck and
> not giving any real feedback on which test failed.
>
> --Bobby
>

[jira] [Resolved] (HADOOP-9298) Cover with unit test package org.apache.hadoop.hdfs.tools

[Vadim Bondarev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Bondarev resolved HADOOP-9298.


Resolution: Duplicate

> Cover with unit test package org.apache.hadoop.hdfs.tools
> -
>
> Key: HADOOP-9298
> URL: https://issues.apache.org/jira/browse/HADOOP-9298
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
>Reporter: Vadim Bondarev
> Attachments: HADOOP-9298-branch-0.23-a.patch, 
> HADOOP-9298-branch-2-a.patch, HADOOP-9298-trunk-a.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-9268) Cover package org.apache.hadoop.hdfs.server.common with tests

[Vadim Bondarev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Bondarev resolved HADOOP-9268.


Resolution: Duplicate

> Cover package org.apache.hadoop.hdfs.server.common  with tests
> --
>
> Key: HADOOP-9268
> URL: https://issues.apache.org/jira/browse/HADOOP-9268
> Project: Hadoop Common
>  Issue Type: Test
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
>Reporter: Vadim Bondarev
> Attachments: HADOOP-9268-branch-0.23-a.patch, 
> HADOOP-9268-branch-2-a.patch, HADOOP-9268-trunk-a.patch
>
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-9314) Cover package org.apache.hadoop.hdfs.server.common with tests

[Vadim Bondarev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vadim Bondarev resolved HADOOP-9314.


Resolution: Duplicate

> Cover package org.apache.hadoop.hdfs.server.common with tests
> -
>
> Key: HADOOP-9314
> URL: https://issues.apache.org/jira/browse/HADOOP-9314
> Project: Hadoop Common
>  Issue Type: Test
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
>Reporter: Vadim Bondarev
>


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira