Created Jira - https://issues.apache.org/jira/browse/DRILL-7030
Kind regards, Arina On Tue, Feb 5, 2019 at 4:58 PM Vitalii Diravka <[email protected]> wrote: > Absolutely agree with Arina. > > I think the core Format Plugins for Parquet, Json and CSV, TSV, PSV files > (which are used for creating Drill tables) can be left in current config > file > and the rest ones should be factored out to the separate config files along > with creating separate modules in Drill *contrib *module. > > Therefore the process of creating the new plugins will be more transparent. > > Kind regards > Vitalii > > > On Tue, Feb 5, 2019 at 3:12 PM Charles Givre <[email protected]> wrote: > > > I’d concur with Arina’s suggestion. I do think this would be useful and > > make it easier to make plugins “pluggable”. > > In the meantime, should we recommend that developers of format-plugins > > include their plugins in the bootstrap-storage-plugins.json? I was > > thinking also that we might want to have some guidelines for unit tests > for > > format plugins. I’m doing some work on the HTTPD format plugin and found > > some issues which cause it to throw NPEs. > > — C > > > > > > > On Feb 5, 2019, at 06:40, Arina Yelchiyeva <[email protected] > > > > wrote: > > > > > > Hi all, > > > > > > Before we were adding new formats / plugins into the exec module. > > Eventually we came up to the point that exec package size is growing and > > adding plugin and format contributions is better to separate out in the > > different module. > > > Now we have contrib module where we add such contributions. Plugins are > > pluggable, there are added automatically by means of having > > drill-module.conf file which points to the scanning packages. > > > Format plugins are using the same approach, the only problem is that > > they are not added into bootstrap-storage-plugins.json. So when adding > new > > format plugin, in order for it to automatically appear in Drill Web UI, > > developer has to update bootstrap file which is in the exec module. > > > My suggestion we implement some functionality that would merge format > > config with the bootstrap one. For example, each plugin would have to > have > > bootstrap-format.json file with the information to which plugin format > > should be added (structure the same as in > bootstrap-storage-plugins.json): > > > Example: > > > > > > { > > > "storage":{ > > > dfs: { > > > formats: { > > > "psv" : { > > > type: "msgpack", > > > extensions: [ "mp" ] > > > } > > > } > > > } > > > } > > > } > > > > > > Then during Drill start up such bootstrap-format.json files will be > > merged with bootstrap-storage-plugins.json. > > > > > > > > > Current open PR for adding new format plugins: > > > Format plugin for LTSV files - > https://github.com/apache/drill/pull/1627 > > > SYSLOG (RFC-5424) Format Plugin - > > https://github.com/apache/drill/pull/1530 > > > Msgpack format reader - https://github.com/apache/drill/pull/1500 > > > > > > Any suggestions? > > > > > > Kind regards, > > > Arina > > > > >
