Jesus Camacho Rodriguez created HIVE-19732:
----------------------------------------------
Summary: Create utility method to set staging tmp directory
correctly
Key: HIVE-19732
URL: https://issues.apache.org/jira/browse/HIVE-19732
Project: Hive
Issue Type: Test
Components: Test
Reporter: Jesus Camacho Rodriguez
Many tests are flaky because they point to same default staging tmp directory.
For instance:
{noformat}
org.apache.hadoop.util.Shell$ExitCodeException: chmod: cannot access
‘/tmp/hadoop/mapred/staging/hiveptest985275899/.staging/job_local985275899_0088’:
No such file or directory
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1009)
~[hadoop-common-3.1.0.jar:?]
at org.apache.hadoop.util.Shell.run(Shell.java:902)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1227)
~[hadoop-common-3.1.0.jar:?]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1321)
~[hadoop-common-3.1.0.jar:?]
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1303)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:840)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:508)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:489)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:511)
~[hadoop-common-3.1.0.jar:?]
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:727)
~[hadoop-common-3.1.0.jar:?]
at
org.apache.hadoop.mapreduce.JobResourceUploader.mkdirs(JobResourceUploader.java:658)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at
org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:172)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at
org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:133)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:102)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:197)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_102]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_102]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
~[hadoop-common-3.1.0.jar:?]
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
~[hadoop-mapreduce-client-core-3.1.0.jar:?]
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
[hadoop-mapreduce-client-core-3.1.0.jar:?]
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_102]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_102]
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
[pig-0.16.0-h2.jar:?]
at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:194)
[pig-0.16.0-h2.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276)
[pig-0.16.0-h2.jar:?]
{noformat}
Till now, those tests have been fixed individually. It would be useful to
create a utility method that given the conf object and the test name, sets the
correct property values correctly so staging directories do not clash.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
