Boaz Ben-Zvi created DRILL-7069: ----------------------------------- Summary: Poor performance of transformBinaryInMetadataCache Key: DRILL-7069 URL: https://issues.apache.org/jira/browse/DRILL-7069 Project: Apache Drill Issue Type: Improvement Components: Metadata Affects Versions: 1.15.0 Reporter: Boaz Ben-Zvi Assignee: Boaz Ben-Zvi Fix For: 1.16.0
The performance of the method *transformBinaryInMetadataCache* scales poorly as the table's numbers of underlying files, row-groups and columns grow. This method is invoked during planning of every query using this table. A test on a table using 219 directories (each with 20 files), 1 row-group in each file, and 94 columns, measured about *1340 milliseconds*. The main culprit are the version checks, which take place in *every iteration* (i.e., about 400k times in the previous example) and involve construction of 6 MetadataVersion objects (and possibly garbage collections). Removing the version checks from the loops improved this method's performance on the above test down to about *250 milliseconds*. -- This message was sent by Atlassian JIRA (v7.6.3#76005)