Yikun commented on a change in pull request #32867: URL: https://github.com/apache/spark/pull/32867#discussion_r652488153
########## File path: dev/sparktestsupport/modules.py ########## @@ -16,13 +16,65 @@ # from functools import total_ordering +from importlib import import_module +import inspect import itertools import os +from pkgutil import iter_modules import re +import unittest + +from sparktestsupport import SPARK_HOME + all_modules = [] +def _contain_unittests_class(module_name): + """ + Check if the module with specific module_name has classes are derived from unittest.TestCase. + + Such as: + pyspark.tests.test_appsubmit, it will return True, because there is SparkSubmitTests which is + included under the module of pyspark.tests.test_appsubmit, inherits from unittest.TestCase. + `` + + :param module_name: the complete name of module to be checked. + :return: True if contains unittest classes otherwise False. + An ``ModuleNotFoundError`` will raise if the module is not found + """ + _module = import_module(module_name) Review comment: @HyukjinKwon the module import based discoverd depends on pyspark and serveral dependency of pyspark, so it's a bit hard to discover the modules. Do you have any idea on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org