[jira] [Commented] (HADOOP-9267) hadoop -help, -h, --help should show usage instructions
[
https://issues.apache.org/jira/browse/HADOOP-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585087#comment-13585087
]
Hudson commented on HADOOP-9267:
Integrated in Hadoop-Yarn-trunk #136 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/136/])
HADOOP-9267. hadoop -help, -h, --help should show usage instructions.
Contributed by Andrew Wang. (Revision 1449161)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1449161
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
* /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/bin/yarn
hadoop -help, -h, --help should show usage instructions
---
Key: HADOOP-9267
URL: https://issues.apache.org/jira/browse/HADOOP-9267
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
Fix For: 2.0.4-beta
Attachments: hadoop-9267-1.patch, hadoop-9267-2.patch
It's not friendly for new users when the command line scripts don't show
usage instructions when passed the defacto Unix usage flags. Imagine this
sequence of commands:
{noformat}
- % hadoop --help
Error: No command named `--help' was found. Perhaps you meant `hadoop -help'
- % hadoop -help
Error: No command named `-help' was found. Perhaps you meant `hadoop help'
- % hadoop help
Exception in thread main java.lang.NoClassDefFoundError: help
Caused by: java.lang.ClassNotFoundException: help
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: help. Program will exit.
{noformat}
Same applies for the `hdfs` script.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9267) hadoop -help, -h, --help should show usage instructions
[
https://issues.apache.org/jira/browse/HADOOP-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585111#comment-13585111
]
Hudson commented on HADOOP-9267:
Integrated in Hadoop-Hdfs-trunk #1325 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1325/])
HADOOP-9267. hadoop -help, -h, --help should show usage instructions.
Contributed by Andrew Wang. (Revision 1449161)
Result = SUCCESS
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1449161
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
* /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/bin/yarn
hadoop -help, -h, --help should show usage instructions
---
Key: HADOOP-9267
URL: https://issues.apache.org/jira/browse/HADOOP-9267
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
Fix For: 2.0.4-beta
Attachments: hadoop-9267-1.patch, hadoop-9267-2.patch
It's not friendly for new users when the command line scripts don't show
usage instructions when passed the defacto Unix usage flags. Imagine this
sequence of commands:
{noformat}
- % hadoop --help
Error: No command named `--help' was found. Perhaps you meant `hadoop -help'
- % hadoop -help
Error: No command named `-help' was found. Perhaps you meant `hadoop help'
- % hadoop help
Exception in thread main java.lang.NoClassDefFoundError: help
Caused by: java.lang.ClassNotFoundException: help
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: help. Program will exit.
{noformat}
Same applies for the `hdfs` script.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9328) INSERT INTO a S3 external table with no reduce phase results in FileNotFoundException
[ https://issues.apache.org/jira/browse/HADOOP-9328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585113#comment-13585113 ] Steve Loughran commented on HADOOP-9328: Does this work on Hadoop 1.x? INSERT INTO a S3 external table with no reduce phase results in FileNotFoundException - Key: HADOOP-9328 URL: https://issues.apache.org/jira/browse/HADOOP-9328 Project: Hadoop Common Issue Type: Bug Affects Versions: 0.9.0 Environment: YARN, Hadoop 2.0.2-alpha Ubuntu Reporter: Marc Limotte With Yarn and Hadoop 2.0.2-alpha, hive 0.9.0. The destination is an S3 table, the source for the query is a small hive managed table. CREATE EXTERNAL TABLE payout_state_product ( state STRING, product_id STRING, element_id INT, element_value DOUBLE, number_of_fields INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION 's3://com.weatherbill.foo/bar/payout_state_product/'; A simple query to copy the results from the hive managed table into a S3. hive INSERT OVERWRITE TABLE payout_state_product SELECT * FROM payout_state_product_cached; Total MapReduce jobs = 2 Launching Job 1 out of 2 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_1360884012490_0014, Tracking URL = http://i-9ff9e9ef.us-east-1.production.climatedna.net:8088/proxy/application_1360884012490_0014/ Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=i-9ff9e9ef.us-east-1.production.climatedna.net:8032 -kill job_1360884012490_0014 Hadoop job information for Stage-1: number of mappers: 100; number of reducers: 0 2013-02-22 19:15:46,709 Stage-1 map = 0%, reduce = 0% ...snip... 2013-02-22 19:17:02,374 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 427.13 sec MapReduce Total cumulative CPU time: 7 minutes 7 seconds 130 msec Ended Job = job_1360884012490_0014 Ended Job = -1776780875, job is filtered out (removed at runtime). Launching Job 2 out of 2 Number of reduce tasks is set to 0 since there's no reduce operator java.io.FileNotFoundException: File does not exist: /tmp/hive-marc/hive_2013-02-22_19-15-31_691_7365912335285010827/-ext-10002/00_0 at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:782) at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat$OneFileInfo.init(CombineFileInputFormat.java:493) at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getMoreSplits(CombineFileInputFormat.java:284) at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:244) at org.apache.hadoop.mapred.lib.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:69) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:386) at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getSplits(HadoopShimsSecure.java:352) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.processPaths(CombineHiveInputFormat.java:419) at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:390) at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:479) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:471) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:366) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435) at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:137) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:134) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1326) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1118) at
[jira] [Resolved] (HADOOP-8419) GzipCodec NPE upon reset with IBM JDK
[ https://issues.apache.org/jira/browse/HADOOP-8419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang resolved HADOOP-8419. --- Resolution: Fixed Hadoop Commons and HDFS trunk builds have been stabilized. Mark this as fixed. GzipCodec NPE upon reset with IBM JDK - Key: HADOOP-8419 URL: https://issues.apache.org/jira/browse/HADOOP-8419 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Li Labels: gzip, ibm-jdk Fix For: 1.1.2 Attachments: HADOOP-8419-branch-1.patch, HADOOP-8419-branch1-v2.patch, HADOOP-8419-trunk.patch, HADOOP-8419-trunk-v2.patch The GzipCodec will NPE upon reset after finish when the native zlib codec is not loaded. When the native zlib is loaded the codec creates a CompressorOutputStream that doesn't have the problem, otherwise, the GZipCodec uses GZIPOutputStream which is extended to provide the resetState method. Since IBM JDK 6 SR9 FP2 including the current JDK 6 SR10, GZIPOutputStream#finish will release the underlying deflater, which causes NPE upon reset. This seems to be an IBM JDK quirk as Sun JDK and OpenJDK doesn't have this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9330) Add custom JUnit4 test runner with configurable timeout
Steve Loughran created HADOOP-9330: -- Summary: Add custom JUnit4 test runner with configurable timeout Key: HADOOP-9330 URL: https://issues.apache.org/jira/browse/HADOOP-9330 Project: Hadoop Common Issue Type: Test Components: test Reporter: Steve Loughran HADOOP-9112 has added a requirement for all new test methods to declare a timeout, so that jenkins/maven builds will have better information on a timeout. Hard coding timeouts into tests is dangerous as it will generate spurious failures on slower machines/networks and when debugging a test. I propose providing a custom JUnit4 test runner that test cases can declare as their test runner; this can provide timeouts specified at run-time, rather than in-source. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9330) Add custom JUnit4 test runner with configurable timeout
[ https://issues.apache.org/jira/browse/HADOOP-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9330: --- Affects Version/s: 3.0.0 Add custom JUnit4 test runner with configurable timeout --- Key: HADOOP-9330 URL: https://issues.apache.org/jira/browse/HADOOP-9330 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0 Reporter: Steve Loughran HADOOP-9112 has added a requirement for all new test methods to declare a timeout, so that jenkins/maven builds will have better information on a timeout. Hard coding timeouts into tests is dangerous as it will generate spurious failures on slower machines/networks and when debugging a test. I propose providing a custom JUnit4 test runner that test cases can declare as their test runner; this can provide timeouts specified at run-time, rather than in-source. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9330) Add custom JUnit4 test runner with configurable timeout
[
https://issues.apache.org/jira/browse/HADOOP-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585192#comment-13585192
]
Steve Loughran commented on HADOOP-9330:
Discussions with the Maven Surefire team
(https://issues.apache.org/jira/browse/HADOOP-9112) imply that we can use the
{{@RunWith}} attribute in a test class to define a custom test runner for it.
This means we can
# Write our own test runner that picks up a default timeout from a system
property.
# Define a base test class, {{HadoopTestBase}}, that declares its use of this.
# Set the maven build up to propagate a timeout via a system property.
# Have jenkins choose a timeout appropriate to it.
Doing this will obviate the need to place brittle test timeouts in source, and
be easy to retrofit to existing test classes.
Assuming all future test cases get built off a new base class (possibly with
tuned YarnTestBase, HdfsTestBase classes), the base classes would have to go
back into branch-1 if there was any goal of backporting new tests from trunk.
Unless Ant can be set up to switch to the new test runner, the {{@RunWith}}
attribute would have to stripped from the backported test cases.
Add custom JUnit4 test runner with configurable timeout
---
Key: HADOOP-9330
URL: https://issues.apache.org/jira/browse/HADOOP-9330
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0
Reporter: Steve Loughran
HADOOP-9112 has added a requirement for all new test methods to declare a
timeout, so that jenkins/maven builds will have better information on a
timeout.
Hard coding timeouts into tests is dangerous as it will generate spurious
failures on slower machines/networks and when debugging a test.
I propose providing a custom JUnit4 test runner that test cases can declare
as their test runner; this can provide timeouts specified at run-time, rather
than in-source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9112) test-patch should -1 for @Tests without a timeout
[ https://issues.apache.org/jira/browse/HADOOP-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585194#comment-13585194 ] Steve Loughran commented on HADOOP-9112: I've added a new JIRA, HADOOP-9330, which proposes the proper way to fix this. As it is, I consider embedded timeouts dangerous and think this change should not be reverted. Certainly it could have been publicised more. test-patch should -1 for @Tests without a timeout - Key: HADOOP-9112 URL: https://issues.apache.org/jira/browse/HADOOP-9112 Project: Hadoop Common Issue Type: Improvement Reporter: Todd Lipcon Assignee: Surenkumar Nihalani Fix For: 3.0.0 Attachments: HADOOP-9112-1.patch, HADOOP-9112-2.patch, HADOOP-9112-3.patch, HADOOP-9112-4.patch, HADOOP-9112-5.patch, HADOOP-9112-6.patch, HADOOP-9112-7.patch With our current test running infrastructure, if a test with no timeout set runs too long, it triggers a surefire-wide timeout, which for some reason doesn't show up as a failed test in the test-patch output. Given that, we should require that all tests have a timeout set, and have test-patch enforce this with a simple check -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9330) Add custom JUnit4 test runner with configurable timeout
[
https://issues.apache.org/jira/browse/HADOOP-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585213#comment-13585213
]
Konstantin Boudnik commented on HADOOP-9330:
As I have commented elsewhere setting a time out in the runtime can be
achievable through {{surefire.timeout}} property, can it not?
Add custom JUnit4 test runner with configurable timeout
---
Key: HADOOP-9330
URL: https://issues.apache.org/jira/browse/HADOOP-9330
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0
Reporter: Steve Loughran
HADOOP-9112 has added a requirement for all new test methods to declare a
timeout, so that jenkins/maven builds will have better information on a
timeout.
Hard coding timeouts into tests is dangerous as it will generate spurious
failures on slower machines/networks and when debugging a test.
I propose providing a custom JUnit4 test runner that test cases can declare
as their test runner; this can provide timeouts specified at run-time, rather
than in-source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9330) Add custom JUnit4 test runner with configurable timeout
[
https://issues.apache.org/jira/browse/HADOOP-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585240#comment-13585240
]
Luke Lu commented on HADOOP-9330:
-
bq. setting a time out in the runtime can be achievable through
surefire.timeout property, can it not?
surefire.timeout is a test process timeout or per test class if fork mode is
always, not _per_ _test_, which is the rationale for HADOOP-9112 in the first
place.
bq. we can use the @RunWith attribute in a test class to define a custom test
runner for it.
There is no need to use a custom test runner. org.junit.rules.Timeout is what
you want. We can create a common base test class like this.
{code}
public class TestBase {
@Rule public Timeout defaultTimeout = new Timeout(Integer.parseInt(
System.getProperty(test.default.timeout, 10)));
...
}
{code}
Add custom JUnit4 test runner with configurable timeout
---
Key: HADOOP-9330
URL: https://issues.apache.org/jira/browse/HADOOP-9330
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0
Reporter: Steve Loughran
HADOOP-9112 has added a requirement for all new test methods to declare a
timeout, so that jenkins/maven builds will have better information on a
timeout.
Hard coding timeouts into tests is dangerous as it will generate spurious
failures on slower machines/networks and when debugging a test.
I propose providing a custom JUnit4 test runner that test cases can declare
as their test runner; this can provide timeouts specified at run-time, rather
than in-source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9112) test-patch should -1 for @Tests without a timeout
[ https://issues.apache.org/jira/browse/HADOOP-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585243#comment-13585243 ] Luke Lu commented on HADOOP-9112: - We had a timeout problem. We added regex to enforce per test timeout. Now we have two problems. :) Seriously, IMO, we should use org.junit.rules.Timeout in a base test class and be done with it. test-patch should -1 for @Tests without a timeout - Key: HADOOP-9112 URL: https://issues.apache.org/jira/browse/HADOOP-9112 Project: Hadoop Common Issue Type: Improvement Reporter: Todd Lipcon Assignee: Surenkumar Nihalani Fix For: 3.0.0 Attachments: HADOOP-9112-1.patch, HADOOP-9112-2.patch, HADOOP-9112-3.patch, HADOOP-9112-4.patch, HADOOP-9112-5.patch, HADOOP-9112-6.patch, HADOOP-9112-7.patch With our current test running infrastructure, if a test with no timeout set runs too long, it triggers a surefire-wide timeout, which for some reason doesn't show up as a failed test in the test-patch output. Given that, we should require that all tests have a timeout set, and have test-patch enforce this with a simple check -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9117) replace protoc ant plugin exec with a maven plugin
[
https://issues.apache.org/jira/browse/HADOOP-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585245#comment-13585245
]
Tsz Wo (Nicholas), SZE commented on HADOOP-9117:
Yes, it is an Eclipse only problem. Thanks Chris to add that.
... changing the binding phase of the protoc plugin to 'process-resources'
...
How to do it exactly? I tried the following changes but it did not seem to
work.
{code}
Index: hadoop-common-project/hadoop-common/pom.xml
===
--- hadoop-common-project/hadoop-common/pom.xml (revision 1449265)
+++ hadoop-common-project/hadoop-common/pom.xml (working copy)
@@ -279,7 +279,7 @@
executions
execution
idversion-info/id
-phasegenerate-resources/phase
+phaseprocess-resources/phase
goals
goalversion-info/goal
/goals
@@ -295,7 +295,7 @@
/execution
execution
idcompile-protoc/id
-phasegenerate-sources/phase
+phaseprocess-resources/phase
goals
goalprotoc/goal
/goals
@@ -320,7 +320,7 @@
/execution
execution
idcompile-test-protoc/id
-phasegenerate-test-sources/phase
+phaseprocess-resources/phase
goals
goalprotoc/goal
/goals
{code}
replace protoc ant plugin exec with a maven plugin
--
Key: HADOOP-9117
URL: https://issues.apache.org/jira/browse/HADOOP-9117
Project: Hadoop Common
Issue Type: Improvement
Components: build
Affects Versions: 2.0.2-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.0.4-beta
Attachments: HADOOP-9117.patch, HADOOP-9117.patch, HADOOP-9117.patch,
HADOOP-9117.patch, HADOOP-9117.patch, HADOOP-9117.patch
The protoc compiler is currently invoked using ant plugin exec. There is a
bug in the ant plugin exec task which does not consume the STDOUT or STDERR
appropriately making the build to stop sometimes (you need to press enter to
continue).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7487) DF should throw a more reasonable exception when mount cannot be determined
[
https://issues.apache.org/jira/browse/HADOOP-7487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-7487:
Attachment: hadoop-7487-2.patch
Thanks for the review Aaron!
I added the checks on the expected exceptions, and reran {{TestDFVariation}}
and {{TestNamenodeCapacityReport}} successfully.
DF should throw a more reasonable exception when mount cannot be determined
---
Key: HADOOP-7487
URL: https://issues.apache.org/jira/browse/HADOOP-7487
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Andrew Wang
Labels: noob
Attachments: hadoop-7487-1.patch, hadoop-7487-2.patch
Currently, when using the DF class to determine the mount corresponding to a
given directory, it will throw the generic exception Expecting a line not
the end of stream if it can't determine the mount (for example if the
directory doesn't exist).
This error message should be improved in several ways:
# If the dir to check doesn't exist, we can see that before even execing df,
and throw a better exception (or behave better by chopping path components
until it exists)
# Rather than parsing the lines out of df's stdout, collect the whole output,
and then parse. So, if df returns a non-zero exit code, we can avoid trying
to parse the empty result
# If there's a success exit code, and we still can't parse it (eg
incompatible OS), we should include the unparseable line in the exception
message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7487) DF should throw a more reasonable exception when mount cannot be determined
[ https://issues.apache.org/jira/browse/HADOOP-7487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-7487: Affects Version/s: 3.0.0 2.0.3-alpha Status: Patch Available (was: Open) DF should throw a more reasonable exception when mount cannot be determined --- Key: HADOOP-7487 URL: https://issues.apache.org/jira/browse/HADOOP-7487 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.0.3-alpha, 0.23.0, 3.0.0 Reporter: Todd Lipcon Assignee: Andrew Wang Labels: noob Attachments: hadoop-7487-1.patch, hadoop-7487-2.patch Currently, when using the DF class to determine the mount corresponding to a given directory, it will throw the generic exception Expecting a line not the end of stream if it can't determine the mount (for example if the directory doesn't exist). This error message should be improved in several ways: # If the dir to check doesn't exist, we can see that before even execing df, and throw a better exception (or behave better by chopping path components until it exists) # Rather than parsing the lines out of df's stdout, collect the whole output, and then parse. So, if df returns a non-zero exit code, we can avoid trying to parse the empty result # If there's a success exit code, and we still can't parse it (eg incompatible OS), we should include the unparseable line in the exception message. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7487) DF should throw a more reasonable exception when mount cannot be determined
[
https://issues.apache.org/jira/browse/HADOOP-7487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585266#comment-13585266
]
Hadoop QA commented on HADOOP-7487:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12570635/hadoop-7487-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 one of tests included doesn't have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2223//console
This message is automatically generated.
DF should throw a more reasonable exception when mount cannot be determined
---
Key: HADOOP-7487
URL: https://issues.apache.org/jira/browse/HADOOP-7487
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 0.23.0, 3.0.0, 2.0.3-alpha
Reporter: Todd Lipcon
Assignee: Andrew Wang
Labels: noob
Attachments: hadoop-7487-1.patch, hadoop-7487-2.patch
Currently, when using the DF class to determine the mount corresponding to a
given directory, it will throw the generic exception Expecting a line not
the end of stream if it can't determine the mount (for example if the
directory doesn't exist).
This error message should be improved in several ways:
# If the dir to check doesn't exist, we can see that before even execing df,
and throw a better exception (or behave better by chopping path components
until it exists)
# Rather than parsing the lines out of df's stdout, collect the whole output,
and then parse. So, if df returns a non-zero exit code, we can avoid trying
to parse the empty result
# If there's a success exit code, and we still can't parse it (eg
incompatible OS), we should include the unparseable line in the exception
message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9330) Add custom JUnit4 test runner with configurable timeout
[ https://issues.apache.org/jira/browse/HADOOP-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585291#comment-13585291 ] Konstantin Boudnik commented on HADOOP-9330: Avoiding subclassing of a common test class is one of the main reason people are migrating away from Junit3, IMO. Do we really want to enforce subclassing? As for 'per class' vs 'per testcase': if your test case has timed out then it will terminate the whole class run, won't it? Add custom JUnit4 test runner with configurable timeout --- Key: HADOOP-9330 URL: https://issues.apache.org/jira/browse/HADOOP-9330 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0 Reporter: Steve Loughran HADOOP-9112 has added a requirement for all new test methods to declare a timeout, so that jenkins/maven builds will have better information on a timeout. Hard coding timeouts into tests is dangerous as it will generate spurious failures on slower machines/networks and when debugging a test. I propose providing a custom JUnit4 test runner that test cases can declare as their test runner; this can provide timeouts specified at run-time, rather than in-source. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9330) Add custom JUnit4 test runner with configurable timeout
[ https://issues.apache.org/jira/browse/HADOOP-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585300#comment-13585300 ] Luke Lu commented on HADOOP-9330: - bq. Do we really want to enforce subclassing? No, you can use the @Test(timeout=seconds) just fine. You can add the default per test timeout rule to any test class as well. TestBase with a per test default timeout is just a convenience. bq. if your test case has timed out then it will terminate the whole class run, won't it? Please read the comments on HADOOP-9112 to find out why this is not desirable. Add custom JUnit4 test runner with configurable timeout --- Key: HADOOP-9330 URL: https://issues.apache.org/jira/browse/HADOOP-9330 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0 Reporter: Steve Loughran HADOOP-9112 has added a requirement for all new test methods to declare a timeout, so that jenkins/maven builds will have better information on a timeout. Hard coding timeouts into tests is dangerous as it will generate spurious failures on slower machines/networks and when debugging a test. I propose providing a custom JUnit4 test runner that test cases can declare as their test runner; this can provide timeouts specified at run-time, rather than in-source. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9330) Add custom JUnit4 test runner with configurable timeout
[
https://issues.apache.org/jira/browse/HADOOP-9330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13585308#comment-13585308
]
Konstantin Boudnik commented on HADOOP-9330:
Oh right... this is {{@Rule}}. Sorry, I misread your comment. Yes, this sounds
like a proper way to go. Plus, we need to lose the enforcement in the
{{test-patch}} - it is no go in the first place.
Add custom JUnit4 test runner with configurable timeout
---
Key: HADOOP-9330
URL: https://issues.apache.org/jira/browse/HADOOP-9330
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0
Reporter: Steve Loughran
HADOOP-9112 has added a requirement for all new test methods to declare a
timeout, so that jenkins/maven builds will have better information on a
timeout.
Hard coding timeouts into tests is dangerous as it will generate spurious
failures on slower machines/networks and when debugging a test.
I propose providing a custom JUnit4 test runner that test cases can declare
as their test runner; this can provide timeouts specified at run-time, rather
than in-source.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
