zhiping lin created FLINK-22734: ----------------------------------- Summary: ExtractionUtils#getClassReader causes "open too many files" error Key: FLINK-22734 URL: https://issues.apache.org/jira/browse/FLINK-22734 Project: Flink Issue Type: Bug Affects Versions: 1.12.2 Reporter: zhiping lin
when we get classReader in ExtractionUtils {code:java} private static ClassReader getClassReader(Class<?> cls) { final String className = cls.getName().replaceFirst("^.*\\.", "") + ".class"; try { return new ClassReader(cls.getResourceAsStream(className)); } catch (IOException e) { throw new IllegalStateException("Could not instantiate ClassReader.", e); } } {code} we open a inputStream by "cls.getResourceAsStream(className)" and set it as a construct param for ClassReader without any other steps to close it. Also ClassReader's construct method with only a outside inputStream, classReader won't close the inputStream while it finished read the file. This will leads to fd leak with "open too many files" error when we parse a lot independent sqls with different envs in one jvm(the case is that we want to extract the table/data lineage of our flink jobs). {code:java} lsof -p 8011 | wc -l 10534 {code} After checked all files and I found they are opened when loaded the udf jars for each loop to build the env. Then I test this method with try-with-source and it works well. So I wonder if it is necessary to change a little bit just like this {code:java} private static ClassReader getClassReader(Class<?> cls) { final String className = cls.getName().replaceFirst("^.*\\.", "") + ".class"; try (InputStream inputStream = cls.getResourceAsStream(className)) { return new ClassReader(inputStream); } catch (IOException e) { throw new IllegalStateException("Could not instantiate ClassReader.", e); } } {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)