Hi all,
Before we were adding new formats / plugins into the exec module. Eventually we
came up to the point that exec package size is growing and adding plugin and
format contributions is better to separate out in the different module.
Now we have contrib module where we add such contributions. Plugins are
pluggable, there are added automatically by means of having drill-module.conf
file which points to the scanning packages.
Format plugins are using the same approach, the only problem is that they are
not added into bootstrap-storage-plugins.json. So when adding new format
plugin, in order for it to automatically appear in Drill Web UI, developer has
to update bootstrap file which is in the exec module.
My suggestion we implement some functionality that would merge format config
with the bootstrap one. For example, each plugin would have to have
bootstrap-format.json file with the information to which plugin format should
be added (structure the same as in bootstrap-storage-plugins.json):
Example:
{
"storage":{
dfs: {
formats: {
"psv" : {
type: "msgpack",
extensions: [ "mp" ]
}
}
}
}
}
Then during Drill start up such bootstrap-format.json files will be merged with
bootstrap-storage-plugins.json.
Current open PR for adding new format plugins:
Format plugin for LTSV files - https://github.com/apache/drill/pull/1627
SYSLOG (RFC-5424) Format Plugin - https://github.com/apache/drill/pull/1530
Msgpack format reader - https://github.com/apache/drill/pull/1500
Any suggestions?
Kind regards,
Arina
