[
https://issues.apache.org/jira/browse/DRILL-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246851#comment-16246851
]
Paul Rogers commented on DRILL-5949:
------------------------------------
Although, technically, it is quite easy to make this change; backward
compatibility is a challenge because Drill has no mechanism to assist. Here are
two possibilities.
First, assign a priority to the settings as follows:
* Table function options (highest priority)
* Session options
* Plugin options
* System options (lowest priority)
That is, if session options are set, use them. Else, use the plugin options, if
set. Else use the system options (and the system option defaults.) This is
possible because options now identify the scope in which they are set, so we
can differentiate session from system options. The problem here is that the
reader can't actually tell if a setting comes from a table function or from the
plugin definition, so some work may be required to support this pattern.
Second, modify the system/session options to have three values:
{{true}}/{{false}}/{{unset}}. If the value is set to {{unset}}, use the plugin
options. The default option value becomes {{unset}}. If the user changes the
session (or system) option, this is used. So, if a user has changed the system
option, and stored the value in ZK, then that setting will be {{true}} or
{{false}} and will take precedence over the plugin options.
> JSON format options should be part of plugin config; not session options
> ------------------------------------------------------------------------
>
> Key: DRILL-5949
> URL: https://issues.apache.org/jira/browse/DRILL-5949
> Project: Apache Drill
> Issue Type: Improvement
> Affects Versions: 1.12.0
> Reporter: Paul Rogers
>
> Drill provides a JSON record reader. Drill provides two ways to configure
> this reader:
> * Using the JSON plugin configuration.
> * Using a set of session options.
> The plugin configuration defines the file suffix associated with JSON files.
> The session options are:
> * {{store.json.all_text_mode}}
> * {{store.json.read_numbers_as_double}}
> * {{store.json.reader.skip_invalid_records}}
> * {{store.json.reader.print_skipped_invalid_record_number}}
> Suppose I have to JSON files from different sources (and keep them in
> distinct directories.) For the one, I want to use {{all_text_mode}} off as
> the data is nicely formatted. Also, my numbers are fine, so I want
> {{read_numbers_as_double}} off.
> But, the other file is a mess and uses a rather ad-hoc format. So, I want
> these two options turned on.
> As it turns out I often query both files. Today, I must set the session
> options one way to query my "clean" file, then reverse them to query the
> "dirty" file.
> Next, I want to join the two files. How do I set the options one way for the
> "clean" file, and the other for the "dirty" file within the *same query*?
> Can't.
> Now, consider the text format plugin that can read CSV, TSV, PSV and so on.
> It has a variety of options. But, the are *not* session options; they are
> instead options in the plugin definition. This allows me to, say, have a
> plugin config for CSV-with-headers files that I get from source A, and a
> different plugin config for my CSV-without-headers files from source B.
> Suppose we applied the text reader technique to the JSON reader. We'd move
> the session options listed above into the JSON format plugin. Then, I can
> define one plugin for my "clean" files, and a different plugin config for my
> "dirty" files.
> What's more, I can then use table functions to adjust the format for each
> file as needed within a single query. Since table functions are part of a
> query, I can add them to a view that I define for the various JSON files.
> The result is a far simpler user experience than the tedium of resetting
> session options for every query.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
