Hi. We have many test failures due to flakiness on ptest machines; looks like tmp directory is deleted while tests are running:
2018-05-18T10:24:44,991 WARN [Thread-3915] mapred.LocalJobRunner: job_local632888732_0106 java.io.FileNotFoundException: File file:/tmp/hadoop/mapred/staging/hiveptest632888732/.staging/job_local632888 732_0106/job.splitmetainfo does not exist … Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/tmp/temp1540619121/tmp-2080326801 … etc. 1) Can we have tmp directory NOT cleaned up while tests are running? I wonder if it’s easy to nuke between runs. 2) Otherwise we need to weed out all the tests that use tmp and make them not use it. I’m not sure about the best way to do this… hadoop/mapred seems to come from mapreduce.jobtracker.staging.root.dir and hadoop.tmp.dir, but at least after some time looking I cannot find where we set hadoop.tmp.dir to /tmp/hadoop, and it also doesn’t match the default value that has username. Where the other one comes I’m not sure at all. I wonder if it’s viable to deny ptest user access to tmp temporarily, then see what fails at the earliest possible point?