paul-rogers opened a new pull request #1797: DRILL-7278: Refactor result set loader projection mechanism URL: https://github.com/apache/drill/pull/1797 Drill 1.16 added a enhanced scan framework based on the row set mechanisms, and a "provisioned schema" feature build on top of that framework. The next step is to convert the log reader as an example of how to create (or revise) a format plugin to use the new framework. The log reader has two "special" columns that can be projected, but do expand as part of the wildcard. Added a column schema property to enable this feature. Doing this work made clear a bit of refactoring is necessary now that the framework supports: * Wildcard or explicit projection lists * Special columns (no wildcard expansion) in either the "reader" or "provided" schema. * Strict provided schema (reader columns may not be projected in a wildcard if they do not appear in the provided schema.) * Type conversions: implicit, built-in and custom. * Recursively apply all of the above to columns in a may or map array. All of the above is gathered into a new "projection set" abstraction presented to the result set loader. Each time a column is added (either up-front or as the read proceeds), the loader asks "is this column projected?" The projection set uses the above logic to answer the question, and to optionally insert a type conversion. Other changes: * Unifies the type conversion mechanism added as part of provisioned schemas. * Replaced the previous "schema transformer" mechanism with a new "type conversion" mechanism that unifies type conversion, provided schemas and an optional custom type conversion mechanism. * Column writers can report if they are projected. Moved this query from metadata to the column writer itself. * Extended and clarified documentation of the feature. * Revised and/or added unit tests. Once these changes are in, a separate PR will convert the log reader. That work, in turn, will be written up as a tutorial for the developers who want to create or convert plugins to use the new framework.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
