[Piglit] [Patch v2] generated_tests: Catch exception in os.makedirs

[Dylan Baker]

Currently we check for the existence of a directory, and if it doesn't
exist we create it. This would be fine, except that multiple generators
can create the same directory trees and when os.makedirs tries to create
a tree that already exists it raises and OSError exception, with the
errno set to 17 (or errno.EEXIST). Since between the time that a check
is made and the time the os.makedirs completes a second generator can
create the same directory tree we can have exceptions raised
sporadically, especially as the number of threads being used to generate
goes up.

We maintain the check for existence before trying to create the
directories because os.makedirs is slow, so not even trying is better
than trying in vain.

v2: - Use a helper function to reduce code duplication
- use errno.EEXIST instead of magic number 17 (Mark)

Signed-off-by: Dylan Baker dylanx.c.ba...@intel.com
---
 generated_tests/gen_builtin_packing_tests.py   |  4 +-
 generated_tests/gen_builtin_uniform_tests.py   |  5 ++-
 generated_tests/gen_builtin_uniform_tests_fp64.py  |  5 ++-
 generated_tests/gen_const_builtin_equal_tests.py   |  4 +-
 generated_tests/gen_constant_array_size_tests.py   |  4 +-
 .../gen_constant_array_size_tests_fp64.py  |  4 +-
 generated_tests/gen_interpolation_tests.py |  4 +-
 generated_tests/gen_non-lvalue_tests.py|  4 +-
 generated_tests/gen_outerproduct_invalid_params.py |  4 +-
 generated_tests/gen_outerproduct_tests.py  |  4 +-
 generated_tests/gen_shader_bit_encoding_tests.py   |  4 +-
 .../gen_shader_image_load_store_tests.py   |  5 ++-
 generated_tests/gen_texture_lod_tests.py   |  4 +-
 generated_tests/gen_texture_query_lod_tests.py |  4 +-
 generated_tests/gen_uniform_initializer_tests.py   |  7 ++--
 generated_tests/genclbuiltins.py   |  5 ++-
 generated_tests/generate-cl-store-tests.py |  5 ++-
 .../interpolation-qualifier-built-in-variable.py   |  5 +--
 generated_tests/modules/__init__.py|  0
 .../utils.py}  | 48 ++
 20 files changed, 64 insertions(+), 65 deletions(-)
 create mode 100644 generated_tests/modules/__init__.py
 copy generated_tests/{gen_outerproduct_invalid_params.py = modules/utils.py} 
(56%)

diff --git a/generated_tests/gen_builtin_packing_tests.py 
b/generated_tests/gen_builtin_packing_tests.py
index 52bf1a0..901f493 100644
--- a/generated_tests/gen_builtin_packing_tests.py
+++ b/generated_tests/gen_builtin_packing_tests.py
@@ -43,6 +43,7 @@ from math import copysign, fabs, fmod, frexp, isinf, isnan, 
modf
 from numpy import int8, int16, uint8, uint16, uint32, float32
 
 from templates import template_dir
+from modules import utils
 
 TEMPLATES = template_dir(os.path.basename(os.path.splitext(__file__)[0]))
 
@@ -1027,8 +1028,7 @@ class ShaderTest(object):
 
 def write_file(self):
 dirname = os.path.dirname(self.filename)
-if not os.path.exists(dirname):
-os.makedirs(dirname)
+utils.safe_makedirs(dirname)
 
 with open(self.filename, w) as f:
 f.write(self.__template.render(func=self.__func_info))
diff --git a/generated_tests/gen_builtin_uniform_tests.py 
b/generated_tests/gen_builtin_uniform_tests.py
index d89a6f3..9bb3075 100644
--- a/generated_tests/gen_builtin_uniform_tests.py
+++ b/generated_tests/gen_builtin_uniform_tests.py
@@ -52,6 +52,8 @@ import os
 import os.path
 import sys
 
+from modules import utils
+
 
 def compute_offset_and_scale(test_vectors):
 Compute scale and offset values such that for each result in
@@ -537,8 +539,7 @@ class ShaderTest(object):
 shader_test += self.make_test()
 filename = self.filename()
 dirname = os.path.dirname(filename)
-if not os.path.exists(dirname):
-os.makedirs(dirname)
+utils.safe_makedirs(dirname)
 with open(filename, 'w') as f:
 f.write(shader_test)
 
diff --git a/generated_tests/gen_builtin_uniform_tests_fp64.py 
b/generated_tests/gen_builtin_uniform_tests_fp64.py
index ec12650..23032bd 100644
--- a/generated_tests/gen_builtin_uniform_tests_fp64.py
+++ b/generated_tests/gen_builtin_uniform_tests_fp64.py
@@ -52,6 +52,8 @@ import os
 import os.path
 import sys
 
+from modules import utils
+
 
 def compute_offset_and_scale(test_vectors):
 Compute scale and offset values such that for each result in
@@ -507,8 +509,7 @@ class ShaderTest(object):
 shader_test += self.make_test()
 filename = self.filename()
 dirname = os.path.dirname(filename)
-if not os.path.exists(dirname):
-os.makedirs(dirname)
+utils.safe_makedirs(dirname)
 with open(filename, 'w') as f:
 f.write(shader_test)
 
diff --git a/generated_tests/gen_const_builtin_equal_tests.py 
b/generated_tests/gen_const_builtin_equal_tests.py
index eb2856c..47a238c 100644
--- 

Re: [Piglit] [Patch v2] generated_tests: Catch exception in os.makedirs

[Mark Janes]

Thanks, this is a superior implementation imho.

-Mark

Dylan Baker baker.dyla...@gmail.com writes:

 +
 +def safe_makedirs(dirs):
 +A safe function for creating a directory tree.
 +
 +This function wraps os.makedirs, and provides a couple of sanity checks,
 +first, if the directory already exists it doesn't try to create it, and
 +second if the directory comes into existence between the check and 
 creation
 +time (like if another generator creates it) then the exception will be
 +caught.
 +
 +
 +if not os.path.exists(dirs):
 +try:
 +os.makedirs(dirs)
 +except OSError as e:
 +if e.errno == errno.EEXIST:
 +pass
 +raise

___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit