date:20130222


[ 
https://issues.apache.org/jira/browse/HADOOP-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584170#comment-13584170
 ] 

Hudson commented on HADOOP-9112:


Integrated in Hadoop-Yarn-trunk #135 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/135/])
amendment to HADOOP-9112 fix return codes (Surenkumar Nihalani via bobby) 
(Revision 1448745)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1448745
Files : 
* /hadoop/common/trunk/dev-support/test-patch.sh


 test-patch should -1 for @Tests without a timeout
 -

 Key: HADOOP-9112
 URL: https://issues.apache.org/jira/browse/HADOOP-9112
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Todd Lipcon
Assignee: Surenkumar Nihalani
 Fix For: 3.0.0

 Attachments: HADOOP-9112-1.patch, HADOOP-9112-2.patch, 
 HADOOP-9112-3.patch, HADOOP-9112-4.patch, HADOOP-9112-5.patch, 
 HADOOP-9112-6.patch, HADOOP-9112-7.patch


 With our current test running infrastructure, if a test with no timeout set 
 runs too long, it triggers a surefire-wide timeout, which for some reason 
 doesn't show up as a failed test in the test-patch output. Given that, we 
 should require that all tests have a timeout set, and have test-patch enforce 
 this with a simple check

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9325) KerberosAuthenticationHandler AuthenticationFilter and should be able to reference Hadoop configurations

2013-02-22 Thread Kai Zheng (JIRA)

Kai Zheng created HADOOP-9325:
-

 Summary: KerberosAuthenticationHandler AuthenticationFilter and 
should be able to reference Hadoop configurations
 Key: HADOOP-9325
 URL: https://issues.apache.org/jira/browse/HADOOP-9325
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Reporter: Kai Zheng


In KerberosAuthenticationHandler SPNEGO activities, KerberosName is used to get 
short name for client principal, which needs in some Kerberos authentication 
situations to reference translation rules defined in Hadoop configuration file 
like core-site.xml
as follows:

  property
namehadoop.security.auth_to_local/name
value.../value
  /property

Note, this is an issue only if default rule can't meet the requirement and 
custom rules need to be defined.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9112) test-patch should -1 for @Tests without a timeout


[ 
https://issues.apache.org/jira/browse/HADOOP-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584234#comment-13584234
 ] 

Hudson commented on HADOOP-9112:


Integrated in Hadoop-Hdfs-trunk #1324 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1324/])
amendment to HADOOP-9112 fix return codes (Surenkumar Nihalani via bobby) 
(Revision 1448745)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1448745
Files : 
* /hadoop/common/trunk/dev-support/test-patch.sh


 test-patch should -1 for @Tests without a timeout
 -

 Key: HADOOP-9112
 URL: https://issues.apache.org/jira/browse/HADOOP-9112
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Todd Lipcon
Assignee: Surenkumar Nihalani
 Fix For: 3.0.0

 Attachments: HADOOP-9112-1.patch, HADOOP-9112-2.patch, 
 HADOOP-9112-3.patch, HADOOP-9112-4.patch, HADOOP-9112-5.patch, 
 HADOOP-9112-6.patch, HADOOP-9112-7.patch


 With our current test running infrastructure, if a test with no timeout set 
 runs too long, it triggers a surefire-wide timeout, which for some reason 
 doesn't show up as a failed test in the test-patch output. Given that, we 
 should require that all tests have a timeout set, and have test-patch enforce 
 this with a simple check

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9112) test-patch should -1 for @Tests without a timeout


[ 
https://issues.apache.org/jira/browse/HADOOP-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584286#comment-13584286
 ] 

Hudson commented on HADOOP-9112:


Integrated in Hadoop-Mapreduce-trunk #1352 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1352/])
amendment to HADOOP-9112 fix return codes (Surenkumar Nihalani via bobby) 
(Revision 1448745)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1448745
Files : 
* /hadoop/common/trunk/dev-support/test-patch.sh


 test-patch should -1 for @Tests without a timeout
 -

 Key: HADOOP-9112
 URL: https://issues.apache.org/jira/browse/HADOOP-9112
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Todd Lipcon
Assignee: Surenkumar Nihalani
 Fix For: 3.0.0

 Attachments: HADOOP-9112-1.patch, HADOOP-9112-2.patch, 
 HADOOP-9112-3.patch, HADOOP-9112-4.patch, HADOOP-9112-5.patch, 
 HADOOP-9112-6.patch, HADOOP-9112-7.patch


 With our current test running infrastructure, if a test with no timeout set 
 runs too long, it triggers a surefire-wide timeout, which for some reason 
 doesn't show up as a failed test in the test-patch output. Given that, we 
 should require that all tests have a timeout set, and have test-patch enforce 
 this with a simple check

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9326) BUG when i run test or when i skipped test and i run package step, i have the same problem

2013-02-22 Thread JLASSI Aymen (JIRA)

JLASSI Aymen created HADOOP-9326:


 Summary: BUG when i run test or when i skipped test and i run 
package step, i have the same problem
 Key: HADOOP-9326
 URL: https://issues.apache.org/jira/browse/HADOOP-9326
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, test
 Environment: For information, i take hadoop with GIT and i run it on 
mac OS (last version)
Reporter: JLASSI Aymen


I'd like to compile hadoop from source code, and when i launch test-step, i 
have the desciption as follows, when i skip the test-step to the package step, 
i have the same problem, the same description of bug:



Results :

Failed tests:   testFailFullyDelete(org.apache.hadoop.fs.TestFileUtil): The 
directory xSubDir *should* not have been deleted. expected:true but 
was:false
  testFailFullyDeleteContents(org.apache.hadoop.fs.TestFileUtil): The directory 
xSubDir *should* not have been deleted. expected:true but was:false
  
testListStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem):
 Should throw IOException
  test0[0](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for 
build/test/temp/RELATIVE1 in 
build/test/temp/RELATIVE0/block4197707426846287299.tmp - FAILED!
  testROBufferDirAndRWBufferDir[0](org.apache.hadoop.fs.TestLocalDirAllocator): 
Checking for build/test/temp/RELATIVE2 in 
build/test/temp/RELATIVE1/block138767728739012230.tmp - FAILED!
  testRWBufferDirBecomesRO[0](org.apache.hadoop.fs.TestLocalDirAllocator): 
Checking for build/test/temp/RELATIVE3 in 
build/test/temp/RELATIVE4/block4888615109050601773.tmp - FAILED!
  test0[1](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for 
/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1
 in 
/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE0/block4663369813226761504.tmp
 - FAILED!
  testROBufferDirAndRWBufferDir[1](org.apache.hadoop.fs.TestLocalDirAllocator): 
Checking for 
/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE2
 in 
/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1/block2846944239985650460.tmp
 - FAILED!
  testRWBufferDirBecomesRO[1](org.apache.hadoop.fs.TestLocalDirAllocator): 
Checking for 
/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE3
 in 
/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE4/block4367331619344952181.tmp
 - FAILED!
  test0[2](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for 
file:/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1
 in 
/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED0/block5687619346377173125.tmp
 - FAILED!
  testROBufferDirAndRWBufferDir[2](org.apache.hadoop.fs.TestLocalDirAllocator): 
Checking for 
file:/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED2
 in 
/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1/block2235209534902942511.tmp
 - FAILED!
  testRWBufferDirBecomesRO[2](org.apache.hadoop.fs.TestLocalDirAllocator): 
Checking for 
file:/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED3
 in 
/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED4/block6994640486900109274.tmp
 - FAILED!
  testReportChecksumFailure(org.apache.hadoop.fs.TestLocalFileSystem)
  
testListStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.viewfs.TestFSMainOperationsLocalFileSystem):
 Should throw IOException
  testCount(org.apache.hadoop.metrics2.util.TestSampleQuantiles): 
expected:50[.00 %ile +/- 5.00%: 1337(..)
  testCheckDir_notDir_local(org.apache.hadoop.util.TestDiskChecker): checkDir 
success
  testCheckDir_notReadable_local(org.apache.hadoop.util.TestDiskChecker): 
checkDir success
  testCheckDir_notWritable_local(org.apache.hadoop.util.TestDiskChecker): 
checkDir success
  testCheckDir_notListable_local(org.apache.hadoop.util.TestDiskChecker): 
checkDir success

Tests run: 1842, Failures: 19, Errors: 0, Skipped: 22

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main  SUCCESS [1.805s]
[INFO]

[jira] [Updated] (HADOOP-9326) maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-common: There a test failures.

2013-02-22 Thread JLASSI Aymen (JIRA)


 [ 
https://issues.apache.org/jira/browse/HADOOP-9326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

JLASSI Aymen updated HADOOP-9326:
-

 Environment: For information, i take hadoop with GIT and i run it on mac 
OS   (was: For information, i take hadoop with GIT and i run it on mac OS (last 
version))
 Summary: maven-surefire-plugin:2.12.3:test (default-test) on project 
hadoop-common: There a test failures.  (was: BUG when i run test or when i 
skipped test and i run package step, i have the same problem)
Hadoop Flags: Incompatible change,Reviewed  (was: Incompatible change)

 maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-common: 
 There a test failures.
 -

 Key: HADOOP-9326
 URL: https://issues.apache.org/jira/browse/HADOOP-9326
 Project: Hadoop Common
  Issue Type: Bug
  Components: build, test
 Environment: For information, i take hadoop with GIT and i run it on 
 mac OS 
Reporter: JLASSI Aymen
   Original Estimate: 336h
  Remaining Estimate: 336h

 I'd like to compile hadoop from source code, and when i launch test-step, i 
 have the desciption as follows, when i skip the test-step to the package 
 step, i have the same problem, the same description of bug:
 Results :
 Failed tests:   testFailFullyDelete(org.apache.hadoop.fs.TestFileUtil): The 
 directory xSubDir *should* not have been deleted. expected:true but 
 was:false
   testFailFullyDeleteContents(org.apache.hadoop.fs.TestFileUtil): The 
 directory xSubDir *should* not have been deleted. expected:true but 
 was:false
   
 testListStatusThrowsExceptionForUnreadableDir(org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem):
  Should throw IOException
   test0[0](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for 
 build/test/temp/RELATIVE1 in 
 build/test/temp/RELATIVE0/block4197707426846287299.tmp - FAILED!
   
 testROBufferDirAndRWBufferDir[0](org.apache.hadoop.fs.TestLocalDirAllocator): 
 Checking for build/test/temp/RELATIVE2 in 
 build/test/temp/RELATIVE1/block138767728739012230.tmp - FAILED!
   testRWBufferDirBecomesRO[0](org.apache.hadoop.fs.TestLocalDirAllocator): 
 Checking for build/test/temp/RELATIVE3 in 
 build/test/temp/RELATIVE4/block4888615109050601773.tmp - FAILED!
   test0[1](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for 
 /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1
  in 
 /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE0/block4663369813226761504.tmp
  - FAILED!
   
 testROBufferDirAndRWBufferDir[1](org.apache.hadoop.fs.TestLocalDirAllocator): 
 Checking for 
 /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE2
  in 
 /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1/block2846944239985650460.tmp
  - FAILED!
   testRWBufferDirBecomesRO[1](org.apache.hadoop.fs.TestLocalDirAllocator): 
 Checking for 
 /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE3
  in 
 /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE4/block4367331619344952181.tmp
  - FAILED!
   test0[2](org.apache.hadoop.fs.TestLocalDirAllocator): Checking for 
 file:/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1
  in 
 /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED0/block5687619346377173125.tmp
  - FAILED!
   
 testROBufferDirAndRWBufferDir[2](org.apache.hadoop.fs.TestLocalDirAllocator): 
 Checking for 
 file:/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED2
  in 
 /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1/block2235209534902942511.tmp
  - FAILED!
   testRWBufferDirBecomesRO[2](org.apache.hadoop.fs.TestLocalDirAllocator): 
 Checking for 
 file:/Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED3
  in 
 /Users/aymenjlassi/Desktop/hadoop_source/releaseGit/hadoop-common/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED4/block6994640486900109274.tmp
  - FAILED!
   testReportChecksumFailure(org.apache.hadoop.fs.TestLocalFileSystem)

[jira] [Created] (HADOOP-9327) Out of date code examples

2013-02-22 Thread Hao Zhong (JIRA)

Hao Zhong created HADOOP-9327:
-

 Summary: Out of date code examples
 Key: HADOOP-9327
 URL: https://issues.apache.org/jira/browse/HADOOP-9327
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Hao Zhong


1. This page contains code examples that use JobConfigurationParser
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/tools/rumen/package-summary.html
JobConfigurationParser jcp = 
  new JobConfigurationParser(interestedProperties);
JobConfigurationParser is deleted in 2.0.3

2. This page contains code examples that use ContextFactory
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics/package-summary.html
 ContextFactory factory = ContextFactory.getFactory();
... examine and/or modify factory attributes ...
MetricsContext context = factory.getContext(myContext);
ContextFactory is deleted in 2.0.3

3. This page contains code examples that use LoggedNetworkTopology
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/tools/rumen/package-summary.html
 do.init(topology.json, conf);

  // get the job summary using TopologyBuilder
  LoggedNetworkTopology topology = topologyBuilder.build();
LoggedNetworkTopology is deleted in 2.0.3

Please revise the documentation to reflect the code.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9325) KerberosAuthenticationHandler AuthenticationFilter and should be able to reference Hadoop configurations

2013-02-22 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584428#comment-13584428
 ] 

Alejandro Abdelnur commented on HADOOP-9325:


Setting the following property should work:

hadoop.http.authentication.kerberos.names.rules=${hadoop.security.auth_to_local}

If it works then we should repurpose this JIRA to update hadoop-auth 
documentation to mention the [PREFIX].kerberos.names.rules property.

And the HttpAuthentication.html page 
(http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/HttpAuthentication.html)
 should also be updated to mention the concrete property in this case (at the 
beginning of this comment). It seems the link from the sidebar in the docs is 
missing the HttpAuthentication.html page, we should add that too.


 KerberosAuthenticationHandler AuthenticationFilter and should be able to 
 reference Hadoop configurations
 

 Key: HADOOP-9325
 URL: https://issues.apache.org/jira/browse/HADOOP-9325
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Reporter: Kai Zheng

 In KerberosAuthenticationHandler SPNEGO activities, KerberosName is used to 
 get short name for client principal, which needs in some Kerberos 
 authentication situations to reference translation rules defined in Hadoop 
 configuration file like core-site.xml
 as follows:
   property
 namehadoop.security.auth_to_local/name
 value.../value
   /property
 Note, this is an issue only if default rule can't meet the requirement and 
 custom rules need to be defined.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9117) replace protoc ant plugin exec with a maven plugin

2013-02-22 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584438#comment-13584438
 ] 

Alejandro Abdelnur commented on HADOOP-9117:


If I'm getting this correct, this seems to be an eclipse issue, right? Can you 
try the following, changing the binding phase of the protoc plugin to 
'process-resources'? While this is not 100% correct, it may do the trick (by 
binding it to a phase that eclipse picks up and is still before the compile 
phase).


 replace protoc ant plugin exec with a maven plugin
 --

 Key: HADOOP-9117
 URL: https://issues.apache.org/jira/browse/HADOOP-9117
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Affects Versions: 2.0.2-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.0.4-beta

 Attachments: HADOOP-9117.patch, HADOOP-9117.patch, HADOOP-9117.patch, 
 HADOOP-9117.patch, HADOOP-9117.patch, HADOOP-9117.patch


 The protoc compiler is currently invoked using ant plugin exec. There is a 
 bug in the ant plugin exec task which does not consume the STDOUT or STDERR 
 appropriately making the build to stop sometimes (you need to press enter to 
 continue).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8562) Enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments

2013-02-22 Thread Chris Nauroth (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584491#comment-13584491
 ] 

Chris Nauroth commented on HADOOP-8562:
---

+1 non-binding

This code has been tested successfully on Linux and Windows for the past 
several months.


 Enhancements to Hadoop for Windows Server and Windows Azure development and 
 runtime environments
 

 Key: HADOOP-8562
 URL: https://issues.apache.org/jira/browse/HADOOP-8562
 Project: Hadoop Common
  Issue Type: New Feature
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: branch-trunk-win.patch, branch-trunk-win.patch, 
 branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, 
 branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, 
 test-untar.tar, test-untar.tgz


 This JIRA tracks the work that needs to be done on trunk to enable Hadoop to 
 run on Windows Server and Azure environments. This incorporates porting 
 relevant work from the similar effort on branch 1 tracked via HADOOP-8079.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9267) hadoop -help, -h, --help should show usage instructions

[
https://issues.apache.org/jira/browse/HADOOP-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584510#comment-13584510
]

Aaron T. Myers commented on HADOOP-9267:

+1, I'm going to commit this momentarily.

hadoop -help, -h, --help should show usage instructions
---

Key: HADOOP-9267
URL: https://issues.apache.org/jira/browse/HADOOP-9267
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
Attachments: hadoop-9267-1.patch, hadoop-9267-2.patch

It's not friendly for new users when the command line scripts don't show
usage instructions when passed the defacto Unix usage flags. Imagine this
sequence of commands:
{noformat}
- % hadoop --help
Error: No command named `--help' was found. Perhaps you meant `hadoop -help'
- % hadoop -help
Error: No command named `-help' was found. Perhaps you meant `hadoop help'
- % hadoop help
Exception in thread main java.lang.NoClassDefFoundError: help
Caused by: java.lang.ClassNotFoundException: help
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: help. Program will exit.
{noformat}
Same applies for the `hdfs` script.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9267) hadoop -help, -h, --help should show usage instructions

[
https://issues.apache.org/jira/browse/HADOOP-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron T. Myers updated HADOOP-9267:
---

Resolution: Fixed
Fix Version/s: 2.0.4-beta
Target Version/s: 2.0.4-beta
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)

I've just committed this to trunk and branch-2.

Thanks a lot for the contribution, Andrew.

hadoop -help, -h, --help should show usage instructions
---

Attachments: hadoop-9267-1.patch, hadoop-9267-2.patch

[jira] [Commented] (HADOOP-9267) hadoop -help, -h, --help should show usage instructions


[ 
https://issues.apache.org/jira/browse/HADOOP-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584531#comment-13584531
 ] 

Hudson commented on HADOOP-9267:


Integrated in Hadoop-trunk-Commit #3376 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/3376/])
HADOOP-9267. hadoop -help, -h, --help should show usage instructions. 
Contributed by Andrew Wang. (Revision 1449161)

 Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1449161
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
* /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/bin/yarn


 hadoop -help, -h, --help should show usage instructions
 ---

 Key: HADOOP-9267
 URL: https://issues.apache.org/jira/browse/HADOOP-9267
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
 Fix For: 2.0.4-beta

 Attachments: hadoop-9267-1.patch, hadoop-9267-2.patch


 It's not friendly for new users when the command line scripts don't show 
 usage instructions when passed the defacto Unix usage flags. Imagine this 
 sequence of commands:
 {noformat}
 - % hadoop --help
 Error: No command named `--help' was found. Perhaps you meant `hadoop -help'
 - % hadoop -help
 Error: No command named `-help' was found. Perhaps you meant `hadoop help'
 - % hadoop help
 Exception in thread main java.lang.NoClassDefFoundError: help
 Caused by: java.lang.ClassNotFoundException: help
   at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 Could not find the main class: help.  Program will exit.
 {noformat}
 Same applies for the `hdfs` script.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-7487) DF should throw a more reasonable exception when mount cannot be determined


[ 
https://issues.apache.org/jira/browse/HADOOP-7487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584592#comment-13584592
 ] 

Aaron T. Myers commented on HADOOP-7487:


Patch looks pretty good to me. One suggestion: In the places in the test where 
you catch expected exceptions, use {{GenericTestUtils#assertExceptionContains}} 
to ensure that you're catching the exception you expect.

 DF should throw a more reasonable exception when mount cannot be determined
 ---

 Key: HADOOP-7487
 URL: https://issues.apache.org/jira/browse/HADOOP-7487
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Andrew Wang
  Labels: noob
 Attachments: hadoop-7487-1.patch


 Currently, when using the DF class to determine the mount corresponding to a 
 given directory, it will throw the generic exception Expecting a line not 
 the end of stream if it can't determine the mount (for example if the 
 directory doesn't exist).
 This error message should be improved in several ways:
 # If the dir to check doesn't exist, we can see that before even execing df, 
 and throw a better exception (or behave better by chopping path components 
 until it exists)
 # Rather than parsing the lines out of df's stdout, collect the whole output, 
 and then parse. So, if df returns a non-zero exit code, we can avoid trying 
 to parse the empty result
 # If there's a success exit code, and we still can't parse it (eg 
 incompatible OS), we should include the unparseable line in the exception 
 message.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-7487) DF should throw a more reasonable exception when mount cannot be determined


[ 
https://issues.apache.org/jira/browse/HADOOP-7487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584593#comment-13584593
 ] 

Aaron T. Myers commented on HADOOP-7487:


I should've said: +1 once this is addressed.

 DF should throw a more reasonable exception when mount cannot be determined
 ---

 Key: HADOOP-7487
 URL: https://issues.apache.org/jira/browse/HADOOP-7487
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Andrew Wang
  Labels: noob
 Attachments: hadoop-7487-1.patch


 Currently, when using the DF class to determine the mount corresponding to a 
 given directory, it will throw the generic exception Expecting a line not 
 the end of stream if it can't determine the mount (for example if the 
 directory doesn't exist).
 This error message should be improved in several ways:
 # If the dir to check doesn't exist, we can see that before even execing df, 
 and throw a better exception (or behave better by chopping path components 
 until it exists)
 # Rather than parsing the lines out of df's stdout, collect the whole output, 
 and then parse. So, if df returns a non-zero exit code, we can avoid trying 
 to parse the empty result
 # If there's a success exit code, and we still can't parse it (eg 
 incompatible OS), we should include the unparseable line in the exception 
 message.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8569) CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE


 [ 
https://issues.apache.org/jira/browse/HADOOP-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers updated HADOOP-8569:
---

Target Version/s: 2.0.4-beta
  Status: Open  (was: Patch Available)

Looks like the patch no longer applies to trunk. Colin, mind posting an updated 
patch?

I personally find Colin's arguments compelling - there doesn't seem to be much 
if any downside, and adding this really just serves to restore the previous 
situation before introduction of cmake. If there are no further objections, 
I'll commit this once Colin posts an updated patch.

 CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE
 

 Key: HADOOP-8569
 URL: https://issues.apache.org/jira/browse/HADOOP-8569
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8569.001.patch


 In the native code, we should define _GNU_SOURCE and _LARGEFILE_SOURCE so 
 that all of the functions on Linux are available.
 _LARGEFILE enables fseeko and ftello; _GNU_SOURCE enables a variety of 
 Linux-specific functions from glibc, including sync_file_range.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9318) when exiting on a signal, print the signal name first


 [ 
https://issues.apache.org/jira/browse/HADOOP-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers updated HADOOP-9318:
---

Status: Open  (was: Patch Available)

Canceling patch so Colin can address Suresh's feedback.

 when exiting on a signal, print the signal name first
 -

 Key: HADOOP-9318
 URL: https://issues.apache.org/jira/browse/HADOOP-9318
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-9318.001.patch


 On UNIX, it would be nice to know when a Hadoop daemon had exited on a 
 signal.  For example, if a daemon exited because the system administrator 
 sent SIGTERM (i.e. {{killall java}}), it would be nice to know that.  
 Although some of this can be deduced from context and {{SHUTDOWN_MSG}}, it 
 would be nice to have it be explicit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9279) Compile failure when hadoop-maven-plugins doesn't exist in local repository

2013-02-22 Thread Suresh Srinivas (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584712#comment-13584712
 ] 

Suresh Srinivas commented on HADOOP-9279:
-

+1 for the patch.

[~tucu00] Can you please do a quick review to see if you concerns are 
addressed. If I do not hearback, I will commit this change this weekend.

 Compile failure when hadoop-maven-plugins doesn't exist in local repository
 ---

 Key: HADOOP-9279
 URL: https://issues.apache.org/jira/browse/HADOOP-9279
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build, documentation
Affects Versions: 3.0.0
 Environment: hadoop-maven-plugins-3.0.0-SNAPSHOT.jar doesn't exist in 
 local repository or at maven central repository.
Reporter: Tsuyoshi OZAWA
Assignee: Tsuyoshi OZAWA
 Attachments: HADOOP-9279.2.patch, HADOOP-9279.3.patch, 
 HADOOP-9279.patch


 In current hadoop-trunk, compile fails when 
 hadoop-maven-plugins-3.0.0-SNAPSHOT.jar doesn't exist in local repository or 
 at maven central repository.
 The affected components are:
 ./hadoop-common-project/hadoop-common/pom.xml
 ./hadoop-maven-plugins/pom.xml
 ./hadoop-project/pom.xml
 ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
 The error log is as follows
 {quote}
 [ERROR] Plugin org.apache.hadoop.maven.plugin:hadoop-maven-plugins:1.0 or one 
 of its dependencies could not be resolved: Failed to read artifact descriptor 
 for org.apache.hadoop.maven.plugin:hadoop-maven-plugins:jar:1.0: Could not 
 find artifact org.apache.hadoop.maven.plugin:hadoop-maven-plugins:pom:1.0 in 
 central (http://repo.maven.apache.org/maven2) - [Help 1]
 {quote}
 This can be avoidable if hadoop-maven-plugins is installed before packaging. 
 This is undocumented, so this should get documented.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-7487) DF should throw a more reasonable exception when mount cannot be determined


 [ 
https://issues.apache.org/jira/browse/HADOOP-7487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers updated HADOOP-7487:
---

Status: Open  (was: Patch Available)

 DF should throw a more reasonable exception when mount cannot be determined
 ---

 Key: HADOOP-7487
 URL: https://issues.apache.org/jira/browse/HADOOP-7487
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Andrew Wang
  Labels: noob
 Attachments: hadoop-7487-1.patch


 Currently, when using the DF class to determine the mount corresponding to a 
 given directory, it will throw the generic exception Expecting a line not 
 the end of stream if it can't determine the mount (for example if the 
 directory doesn't exist).
 This error message should be improved in several ways:
 # If the dir to check doesn't exist, we can see that before even execing df, 
 and throw a better exception (or behave better by chopping path components 
 until it exists)
 # Rather than parsing the lines out of df's stdout, collect the whole output, 
 and then parse. So, if df returns a non-zero exit code, we can avoid trying 
 to parse the empty result
 # If there's a success exit code, and we still can't parse it (eg 
 incompatible OS), we should include the unparseable line in the exception 
 message.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9328) INSERT INTO a S3 external table with no reduce phase results in FileNotFoundException

2013-02-22 Thread Marc Limotte (JIRA)

Marc Limotte created HADOOP-9328:


 Summary: INSERT INTO a S3 external table with no reduce phase 
results in FileNotFoundException
 Key: HADOOP-9328
 URL: https://issues.apache.org/jira/browse/HADOOP-9328
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.9.0
 Environment: YARN, Hadoop 2.0.2-alpha
Ubuntu
Reporter: Marc Limotte


With Yarn and Hadoop 2.0.2-alpha, hive 0.9.0.

The destination is an S3 table, the source for the query is a small hive 
managed table.

CREATE EXTERNAL TABLE payout_state_product (
  state STRING,
  product_id STRING,
  element_id INT,
  element_value DOUBLE,
  number_of_fields INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 's3://com.weatherbill.foo/bar/payout_state_product/';

A simple query to copy the results from the hive managed table into a S3. 

hive INSERT OVERWRITE TABLE payout_state_product 
SELECT * FROM payout_state_product_cached; 

Total MapReduce jobs = 2 
Launching Job 1 out of 2 
Number of reduce tasks is set to 0 since there's no reduce operator 
Starting Job = job_1360884012490_0014, Tracking URL = 
http://i-9ff9e9ef.us-east-1.production.climatedna.net:8088/proxy/application_1360884012490_0014/
 
Kill Command = /usr/lib/hadoop/bin/hadoop job 
-Dmapred.job.tracker=i-9ff9e9ef.us-east-1.production.climatedna.net:8032 -kill 
job_1360884012490_0014 
Hadoop job information for Stage-1: number of mappers: 100; number of reducers: 
0 
2013-02-22 19:15:46,709 Stage-1 map = 0%, reduce = 0% 
...snip... 
2013-02-22 19:17:02,374 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 427.13 
sec 
MapReduce Total cumulative CPU time: 7 minutes 7 seconds 130 msec 
Ended Job = job_1360884012490_0014 
Ended Job = -1776780875, job is filtered out (removed at runtime). 
Launching Job 2 out of 2 
Number of reduce tasks is set to 0 since there's no reduce operator 
java.io.FileNotFoundException: File does not exist: 
/tmp/hive-marc/hive_2013-02-22_19-15-31_691_7365912335285010827/-ext-10002/00_0
 
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:782)
 
at 
org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat$OneFileInfo.init(CombineFileInputFormat.java:493)
 
at 
org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:284)
 
at 
org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:244)
 
at 
org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:69)
 
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:386)
 
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:352)
 
at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.processPaths(CombineHiveInputFormat.java:419)
 
at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:390)
 
at 
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:479) 
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) 
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366)
 
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) 
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:396) 
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
 
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215) 
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617) 
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:396) 
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
 
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612) 
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435) 
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:137) 
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:134) 
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) 
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1326) 
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1118) 
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951) 
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258) 
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:215) 
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406) 
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:689) 
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:557) 
at

[jira] [Commented] (HADOOP-9328) INSERT INTO a S3 external table with no reduce phase results in FileNotFoundException

2013-02-22 Thread Marc Limotte (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584753#comment-13584753
 ] 

Marc Limotte commented on HADOOP-9328:
--

I also tried explicitly listing the columns instead of '*', but that still 
failed:
INSERT OVERWRITE TABLE payout_state_product 
SELECT * FROM payout_state_product_cached;
-- FAILS

I tried this multiple times, fails every time.  But I have a bunch of other 
queries that work and can write to S3.  The distinguishing factor for this 
query is that it has no reduce phase.

Further evidence... the following variation on the query works. It adds a 
simple reduce phase with a pointless group-by:

INSERT OVERWRITE TABLE payout_state_product
SELECT state, product_id, element_id, element_value, number_of_fields 
FROM payout_state_product_cached
GROUP BY state, product_id, element_id, element_value, number_of_fields;

This group by is a feasible work-around, although for large jobs it could be a 
significant performance hit (luckily that's not the case for this particular 
query).

Also, I confirmed that the problem goes away when not writing to S3.  Using a 
variation of the query that writes to a hive managed, hdfs version of the 
destination.


 INSERT INTO a S3 external table with no reduce phase results in 
 FileNotFoundException
 -

 Key: HADOOP-9328
 URL: https://issues.apache.org/jira/browse/HADOOP-9328
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.9.0
 Environment: YARN, Hadoop 2.0.2-alpha
 Ubuntu
Reporter: Marc Limotte

 With Yarn and Hadoop 2.0.2-alpha, hive 0.9.0.
 The destination is an S3 table, the source for the query is a small hive 
 managed table.
 CREATE EXTERNAL TABLE payout_state_product (
   state STRING,
   product_id STRING,
   element_id INT,
   element_value DOUBLE,
   number_of_fields INT)
 ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
 STORED AS TEXTFILE
 LOCATION 's3://com.weatherbill.foo/bar/payout_state_product/';
 A simple query to copy the results from the hive managed table into a S3. 
 hive INSERT OVERWRITE TABLE payout_state_product 
 SELECT * FROM payout_state_product_cached; 
 Total MapReduce jobs = 2 
 Launching Job 1 out of 2 
 Number of reduce tasks is set to 0 since there's no reduce operator 
 Starting Job = job_1360884012490_0014, Tracking URL = 
 http://i-9ff9e9ef.us-east-1.production.climatedna.net:8088/proxy/application_1360884012490_0014/
  
 Kill Command = /usr/lib/hadoop/bin/hadoop job 
 -Dmapred.job.tracker=i-9ff9e9ef.us-east-1.production.climatedna.net:8032 
 -kill job_1360884012490_0014 
 Hadoop job information for Stage-1: number of mappers: 100; number of 
 reducers: 0 
 2013-02-22 19:15:46,709 Stage-1 map = 0%, reduce = 0% 
 ...snip... 
 2013-02-22 19:17:02,374 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 
 427.13 sec 
 MapReduce Total cumulative CPU time: 7 minutes 7 seconds 130 msec 
 Ended Job = job_1360884012490_0014 
 Ended Job = -1776780875, job is filtered out (removed at runtime). 
 Launching Job 2 out of 2 
 Number of reduce tasks is set to 0 since there's no reduce operator 
 java.io.FileNotFoundException: File does not exist: 
 /tmp/hive-marc/hive_2013-02-22_19-15-31_691_7365912335285010827/-ext-10002/00_0
  
 at 
 org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:782)
  
 at 
 org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat$OneFileInfo.init(CombineFileInputFormat.java:493)
  
 at 
 org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:284)
  
 at 
 org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:244)
  
 at 
 org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:69)
  
 at 
 org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:386)
  
 at 
 org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:352)
  
 at 
 org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.processPaths(CombineHiveInputFormat.java:419)
  
 at 
 org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:390)
  
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:479)
  
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) 
 at 
 org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366)
  
 at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) 
 at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) 
 at java.security.AccessController.doPrivileged(Native Method) 
 at javax.security.auth.Subject.doAs(Subject.java:396) 
 at

[jira] [Updated] (HADOOP-8569) CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE


 [ 
https://issues.apache.org/jira/browse/HADOOP-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-8569:
-

Attachment: HADOOP-8569.003.patch

Updated version.

This version leaves out the changes to the YARN CMakeLists.txt, because 
{{_GNU_SOURCE}} was already added there.

 CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE
 

 Key: HADOOP-8569
 URL: https://issues.apache.org/jira/browse/HADOOP-8569
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8569.001.patch, HADOOP-8569.003.patch


 In the native code, we should define _GNU_SOURCE and _LARGEFILE_SOURCE so 
 that all of the functions on Linux are available.
 _LARGEFILE enables fseeko and ftello; _GNU_SOURCE enables a variety of 
 Linux-specific functions from glibc, including sync_file_range.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8569) CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE


[ 
https://issues.apache.org/jira/browse/HADOOP-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584760#comment-13584760
 ] 

Colin Patrick McCabe commented on HADOOP-8569:
--

On a related note, I have a bunch of patches to get native compilation working 
under OpenBSD that I really should submit at some point.

 CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE
 

 Key: HADOOP-8569
 URL: https://issues.apache.org/jira/browse/HADOOP-8569
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8569.001.patch, HADOOP-8569.003.patch


 In the native code, we should define _GNU_SOURCE and _LARGEFILE_SOURCE so 
 that all of the functions on Linux are available.
 _LARGEFILE enables fseeko and ftello; _GNU_SOURCE enables a variety of 
 Linux-specific functions from glibc, including sync_file_range.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8569) CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE


 [ 
https://issues.apache.org/jira/browse/HADOOP-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-8569:
-

Status: Patch Available  (was: Open)

 CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE
 

 Key: HADOOP-8569
 URL: https://issues.apache.org/jira/browse/HADOOP-8569
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8569.001.patch, HADOOP-8569.003.patch


 In the native code, we should define _GNU_SOURCE and _LARGEFILE_SOURCE so 
 that all of the functions on Linux are available.
 _LARGEFILE enables fseeko and ftello; _GNU_SOURCE enables a variety of 
 Linux-specific functions from glibc, including sync_file_range.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9318) when exiting on a signal, print the signal name first


 [ 
https://issues.apache.org/jira/browse/HADOOP-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-9318:
-

Attachment: HADOOP-9318.002.patch

 when exiting on a signal, print the signal name first
 -

 Key: HADOOP-9318
 URL: https://issues.apache.org/jira/browse/HADOOP-9318
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-9318.001.patch, HADOOP-9318.002.patch


 On UNIX, it would be nice to know when a Hadoop daemon had exited on a 
 signal.  For example, if a daemon exited because the system administrator 
 sent SIGTERM (i.e. {{killall java}}), it would be nice to know that.  
 Although some of this can be deduced from context and {{SHUTDOWN_MSG}}, it 
 would be nice to have it be explicit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9318) when exiting on a signal, print the signal name first


 [ 
https://issues.apache.org/jira/browse/HADOOP-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-9318:
-

Status: Patch Available  (was: Open)

 when exiting on a signal, print the signal name first
 -

 Key: HADOOP-9318
 URL: https://issues.apache.org/jira/browse/HADOOP-9318
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-9318.001.patch, HADOOP-9318.002.patch


 On UNIX, it would be nice to know when a Hadoop daemon had exited on a 
 signal.  For example, if a daemon exited because the system administrator 
 sent SIGTERM (i.e. {{killall java}}), it would be nice to know that.  
 Although some of this can be deduced from context and {{SHUTDOWN_MSG}}, it 
 would be nice to have it be explicit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9318) when exiting on a signal, print the signal name first

[
https://issues.apache.org/jira/browse/HADOOP-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584790#comment-13584790
]

Colin Patrick McCabe commented on HADOOP-9318:
--

I added a bunch of javadoc.

bq. Minor nits - in register() method, create StringBuilder after the first
check that throws exception. Optionally, is it better to throw
IllegalStateException instead RTE?

Yeah.

bq. Given the way Handler code is, you just need only a single instance of
Handler. It can be registered in the register method itself using
signal.handle() call right?

Each instance of Handler handles a different signal. Keep in mind that
prevHandler will be different for each signal, so if we combined them all into
one, we'd have to have some kind of Map or something to be able to call the
correct prevHandler. So I think it's easier to leave them separate.

when exiting on a signal, print the signal name first
-

Key: HADOOP-9318
URL: https://issues.apache.org/jira/browse/HADOOP-9318
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.4-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HADOOP-9318.001.patch, HADOOP-9318.002.patch

On UNIX, it would be nice to know when a Hadoop daemon had exited on a
signal. For example, if a daemon exited because the system administrator
sent SIGTERM (i.e. {{killall java}}), it would be nice to know that.
Although some of this can be deduced from context and {{SHUTDOWN_MSG}}, it
would be nice to have it be explicit.

[jira] [Created] (HADOOP-9329) document native build dependencies in BUILDING.txt

Colin Patrick McCabe created HADOOP-9329:


 Summary: document native build dependencies in BUILDING.txt
 Key: HADOOP-9329
 URL: https://issues.apache.org/jira/browse/HADOOP-9329
 Project: Hadoop Common
  Issue Type: Task
Affects Versions: 2.0.4-beta
Reporter: Colin Patrick McCabe
Priority: Trivial


{{BUILDING.txt}} describes {{-Pnative}}, but it does not specify what native 
libraries are needed for the build.  We should address this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-9329) document native build dependencies in BUILDING.txt


 [ 
https://issues.apache.org/jira/browse/HADOOP-9329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-9329:
-

Component/s: documentation

 document native build dependencies in BUILDING.txt
 --

 Key: HADOOP-9329
 URL: https://issues.apache.org/jira/browse/HADOOP-9329
 Project: Hadoop Common
  Issue Type: Task
  Components: documentation
Affects Versions: 2.0.4-beta
Reporter: Colin Patrick McCabe
Priority: Trivial

 {{BUILDING.txt}} describes {{-Pnative}}, but it does not specify what native 
 libraries are needed for the build.  We should address this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9318) when exiting on a signal, print the signal name first


[ 
https://issues.apache.org/jira/browse/HADOOP-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584823#comment-13584823
 ] 

Hadoop QA commented on HADOOP-9318:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12570552/HADOOP-9318.002.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 tests included appear to have a timeout.{color}

  {color:red}-1 javac{color}.  The applied patch generated 1363 javac 
compiler warnings (more than the trunk's current 1356 warnings).

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 5 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2220//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2220//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2220//console

This message is automatically generated.

 when exiting on a signal, print the signal name first
 -

 Key: HADOOP-9318
 URL: https://issues.apache.org/jira/browse/HADOOP-9318
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-9318.001.patch, HADOOP-9318.002.patch


 On UNIX, it would be nice to know when a Hadoop daemon had exited on a 
 signal.  For example, if a daemon exited because the system administrator 
 sent SIGTERM (i.e. {{killall java}}), it would be nice to know that.  
 Although some of this can be deduced from context and {{SHUTDOWN_MSG}}, it 
 would be nice to have it be explicit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-8569) CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE


[ 
https://issues.apache.org/jira/browse/HADOOP-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584866#comment-13584866
 ] 

Hadoop QA commented on HADOOP-8569:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12570546/HADOOP-8569.003.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2219//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2219//console

This message is automatically generated.

 CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE
 

 Key: HADOOP-8569
 URL: https://issues.apache.org/jira/browse/HADOOP-8569
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-8569.001.patch, HADOOP-8569.003.patch


 In the native code, we should define _GNU_SOURCE and _LARGEFILE_SOURCE so 
 that all of the functions on Linux are available.
 _LARGEFILE enables fseeko and ftello; _GNU_SOURCE enables a variety of 
 Linux-specific functions from glibc, including sync_file_range.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded


 [ 
https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Chaiken updated HADOOP-3733:
--

Affects Version/s: 2.0.2-alpha

 s3: URLs break when Secret Key contains a slash, even if encoded
 --

 Key: HADOOP-3733
 URL: https://issues.apache.org/jira/browse/HADOOP-3733
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 0.17.1, 2.0.2-alpha
Reporter: Stuart Sierra
Priority: Minor
 Attachments: hadoop-3733.patch


 When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, 
 distcp fails if the SECRET contains a slash, even when the slash is 
 URL-encoded as %2F.
 Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
 And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
 And your bucket is called mybucket
 You can URL-encode the Secret KKey as 
 Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
 But this doesn't work:
 {noformat}
 $ bin/hadoop distcp file:///source  
 s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
 08/07/09 15:05:22 INFO util.CopyFiles: 
 destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: 
 mybucket
 org.jets3t.service.S3ServiceException: S3 HEAD request failed. 
 ResponseCode=403, ResponseMessage=Forbidden
 at 
 org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
 ...
 With failures, global counters are inaccurate; consider running with -i
 Copy failed: org.apache.hadoop.fs.s3.S3Exception: 
 org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: 
 ?xml version=1.0 
 encoding=UTF-8?ErrorCodeSignatureDoesNotMatch/CodeMessageThe 
 request signature we calculated does not match the signature you provided. 
 Check your key and signing method./Message
 at 
 org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
 ...
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded


 [ 
https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Chaiken updated HADOOP-3733:
--

Attachment: HADOOP-3733.patch

patch for HADOOP-3733 on 2.0.2-alpha


 s3: URLs break when Secret Key contains a slash, even if encoded
 --

 Key: HADOOP-3733
 URL: https://issues.apache.org/jira/browse/HADOOP-3733
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 0.17.1, 2.0.2-alpha
Reporter: Stuart Sierra
Priority: Minor
 Attachments: hadoop-3733.patch, HADOOP-3733.patch


 When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, 
 distcp fails if the SECRET contains a slash, even when the slash is 
 URL-encoded as %2F.
 Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
 And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
 And your bucket is called mybucket
 You can URL-encode the Secret KKey as 
 Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
 But this doesn't work:
 {noformat}
 $ bin/hadoop distcp file:///source  
 s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
 08/07/09 15:05:22 INFO util.CopyFiles: 
 destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: 
 mybucket
 org.jets3t.service.S3ServiceException: S3 HEAD request failed. 
 ResponseCode=403, ResponseMessage=Forbidden
 at 
 org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
 ...
 With failures, global counters are inaccurate; consider running with -i
 Copy failed: org.apache.hadoop.fs.s3.S3Exception: 
 org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: 
 ?xml version=1.0 
 encoding=UTF-8?ErrorCodeSignatureDoesNotMatch/CodeMessageThe 
 request signature we calculated does not match the signature you provided. 
 Check your key and signing method./Message
 at 
 org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
 ...
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9043) winutils can create unusable symlinks

2013-02-22 Thread Chuan Liu (JIRA)

[
https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584877#comment-13584877
]

Chuan Liu commented on HADOOP-9043:
---

To summary the cause of the problem:
Windows symlink creation API - CreateSymbolicLink() - needs to know if the
destination is a file or directory. We called DirectoryCheck() to tell if the
given path is a file or directory. This leads to two issues:

1) The path needs exist in order for the symlink to be created successfully.
This diverges from the Unix behavior, where invalid link can be created
regardless existence of the destination.

2) We have an inconsistency in Windows API in treating forward slashes and
backslashes. In this case, DirectoryCheck() will accept forward slashes and
backslashes when checking if a path is directory or file, while the link
contains forward slashes will become invalid symlinks even if the destination
exists as indicated by DirectoryCheck().

We will not fix the first problem as this roots in how OS handle symlinks and
API differences in symlink creation.

For 2), I think both Arpit's and Ivan's fixes should work. I think Ivan's fix
is safer in general by not considering paths containing forward slashes.

winutils can create unusable symlinks
-

Key: HADOOP-9043
URL: https://issues.apache.org/jira/browse/HADOOP-9043
Project: Hadoop Common
Issue Type: Bug
Components: util
Affects Versions: 1-win, trunk-win
Reporter: Chris Nauroth
Assignee: Arpit Agarwal
Attachments: HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.patch

In general, the winutils symlink command rejects attempts to create symlinks
targeting a destination file that does not exist. However, if given a
symlink destination with forward slashes pointing at a file that does exist,
then it creates the symlink with the forward slashes, and then attempts to
open the file through the symlink will fail.

[jira] [Updated] (HADOOP-9318) when exiting on a signal, print the signal name first


 [ 
https://issues.apache.org/jira/browse/HADOOP-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Patrick McCabe updated HADOOP-9318:
-

Attachment: HADOOP-9318.003.patch

increment {{OK_JAVADOC_WARNINGS}} to take into account the warnings about using 
{{Signal}}, etc.

 when exiting on a signal, print the signal name first
 -

 Key: HADOOP-9318
 URL: https://issues.apache.org/jira/browse/HADOOP-9318
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-9318.001.patch, HADOOP-9318.002.patch, 
 HADOOP-9318.003.patch


 On UNIX, it would be nice to know when a Hadoop daemon had exited on a 
 signal.  For example, if a daemon exited because the system administrator 
 sent SIGTERM (i.e. {{killall java}}), it would be nice to know that.  
 Although some of this can be deduced from context and {{SHUTDOWN_MSG}}, it 
 would be nice to have it be explicit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9318) when exiting on a signal, print the signal name first


[ 
https://issues.apache.org/jira/browse/HADOOP-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584920#comment-13584920
 ] 

Hadoop QA commented on HADOOP-9318:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12570575/HADOOP-9318.003.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified test files.

{color:green}+1 tests included appear to have a timeout.{color}

  {color:red}-1 javac{color}.  The applied patch generated 1363 javac 
compiler warnings (more than the trunk's current 1356 warnings).

{color:red}-1 javadoc{color}.  The javadoc tool appears to have generated 5 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 core tests{color}.  The patch passed unit tests in 
hadoop-common-project/hadoop-common.

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2221//testReport/
Javac warnings: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2221//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: 
https://builds.apache.org/job/PreCommit-HADOOP-Build/2221//console

This message is automatically generated.

 when exiting on a signal, print the signal name first
 -

 Key: HADOOP-9318
 URL: https://issues.apache.org/jira/browse/HADOOP-9318
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.4-beta
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
 Attachments: HADOOP-9318.001.patch, HADOOP-9318.002.patch, 
 HADOOP-9318.003.patch


 On UNIX, it would be nice to know when a Hadoop daemon had exited on a 
 signal.  For example, if a daemon exited because the system administrator 
 sent SIGTERM (i.e. {{killall java}}), it would be nice to know that.  
 Although some of this can be deduced from context and {{SHUTDOWN_MSG}}, it 
 would be nice to have it be explicit.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-9112) test-patch should -1 for @Tests without a timeout

2013-02-22 Thread Konstantin Boudnik (JIRA)


[ 
https://issues.apache.org/jira/browse/HADOOP-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13584958#comment-13584958
 ] 

Konstantin Boudnik commented on HADOOP-9112:


So, why solving a pure Maven issue with policy enforcement that still causing 
head-aches? What's wrong with {{surefire.timeout}} setting? Won't it achieve 
the same essentially without burning everyone with quite an artificial 
requirement to set a timeout on each test?

 test-patch should -1 for @Tests without a timeout
 -

 Key: HADOOP-9112
 URL: https://issues.apache.org/jira/browse/HADOOP-9112
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Todd Lipcon
Assignee: Surenkumar Nihalani
 Fix For: 3.0.0

 Attachments: HADOOP-9112-1.patch, HADOOP-9112-2.patch, 
 HADOOP-9112-3.patch, HADOOP-9112-4.patch, HADOOP-9112-5.patch, 
 HADOOP-9112-6.patch, HADOOP-9112-7.patch


 With our current test running infrastructure, if a test with no timeout set 
 runs too long, it triggers a surefire-wide timeout, which for some reason 
 doesn't show up as a failed test in the test-patch output. Given that, we 
 should require that all tests have a timeout set, and have test-patch enforce 
 this with a simple check

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded


 [ 
https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Chaiken updated HADOOP-3733:
--

Attachment: HADOOP-3733-20130223T011025Z.patch

Added newly-required timeouts to the patch unit tests.  (See HADOOP-9112.)


 s3: URLs break when Secret Key contains a slash, even if encoded
 --

 Key: HADOOP-3733
 URL: https://issues.apache.org/jira/browse/HADOOP-3733
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 0.17.1, 2.0.2-alpha
Reporter: Stuart Sierra
Priority: Minor
 Attachments: HADOOP-3733-20130223T011025Z.patch, hadoop-3733.patch, 
 HADOOP-3733.patch


 When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, 
 distcp fails if the SECRET contains a slash, even when the slash is 
 URL-encoded as %2F.
 Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
 And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
 And your bucket is called mybucket
 You can URL-encode the Secret KKey as 
 Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
 But this doesn't work:
 {noformat}
 $ bin/hadoop distcp file:///source  
 s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
 08/07/09 15:05:22 INFO util.CopyFiles: 
 destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: 
 mybucket
 org.jets3t.service.S3ServiceException: S3 HEAD request failed. 
 ResponseCode=403, ResponseMessage=Forbidden
 at 
 org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
 ...
 With failures, global counters are inaccurate; consider running with -i
 Copy failed: org.apache.hadoop.fs.s3.S3Exception: 
 org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: 
 ?xml version=1.0 
 encoding=UTF-8?ErrorCodeSignatureDoesNotMatch/CodeMessageThe 
 request signature we calculated does not match the signature you provided. 
 Check your key and signing method./Message
 at 
 org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
 ...
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded


 [ 
https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Chaiken updated HADOOP-3733:
--

Release Note: Allows userinfo component of URI authority to contain a slash 
(escaped as %2F).  Especially useful for accessing AWS S3 with distcp or hadoop 
fs.
  Status: Patch Available  (was: Open)

 s3: URLs break when Secret Key contains a slash, even if encoded
 --

 Key: HADOOP-3733
 URL: https://issues.apache.org/jira/browse/HADOOP-3733
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs/s3
Affects Versions: 2.0.2-alpha, 0.17.1
Reporter: Stuart Sierra
Priority: Minor
 Attachments: HADOOP-3733-20130223T011025Z.patch, hadoop-3733.patch, 
 HADOOP-3733.patch


 When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, 
 distcp fails if the SECRET contains a slash, even when the slash is 
 URL-encoded as %2F.
 Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
 And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
 And your bucket is called mybucket
 You can URL-encode the Secret KKey as 
 Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
 But this doesn't work:
 {noformat}
 $ bin/hadoop distcp file:///source  
 s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
 08/07/09 15:05:22 INFO util.CopyFiles: 
 destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: 
 mybucket
 org.jets3t.service.S3ServiceException: S3 HEAD request failed. 
 ResponseCode=403, ResponseMessage=Forbidden
 at 
 org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
 ...
 With failures, global counters are inaccurate; consider running with -i
 Copy failed: org.apache.hadoop.fs.s3.S3Exception: 
 org.jets3t.service.S3ServiceException: S3 PUT failed. XML Error Message: 
 ?xml version=1.0 
 encoding=UTF-8?ErrorCodeSignatureDoesNotMatch/CodeMessageThe 
 request signature we calculated does not match the signature you provided. 
 Check your key and signing method./Message
 at 
 org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
 ...
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HADOOP-3733) s3: URLs break when Secret Key contains a slash, even if encoded