Jean-Claude, this is a good question. Previously new formats were added into the exec module and thus modifying bootstrap-storage-plugins.json was making sense. Since now we are adding new format plugins into the contrib directory, this raises the question on how to publish new format in dfs storage plugin. In StoragePluginRegistryImpl if we find several bootstrap-storage-plugins.json we load data from all of them but if there is a duplicate between storage plugins (ex: dfs), the duplicate would be ignored. Basically, we need to implement new mechanism that would merge two storage plugins formats.
P.S. 1. There is a difference between storage plugin and storage format, I believe you have implemented the second. 2. I suggest we put all formats under contrib/format/.., so they are not confused with plugins. Kind regards, Arina On Thu, Sep 27, 2018 at 12:54 PM Vitalii Diravka <vita...@apache.org> wrote: > I'm not sure what you exactly mean. All *bootstrap-storage-plugins.json* > are registered and stored in Persistent Store > for the first fresh instantiating of Drill, see > *StoragePluginRegistryImpl.loadBootstrapPlugins() > *for details. > > I have noticed that you have a typo in the > *bootstrap-storage-plugins.json *file > name (boostrap -> *bootstrap*). > > Also you can use *storage-plugins-override.conf *for configuring plugins > configs during start-up [1]. > > [1] > > https://drill.apache.org/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-override.conf-file > > > On Wed, Sep 26, 2018 at 10:17 PM Jean-Claude Cote <jcc...@gmail.com> > wrote: > > > I see how the cluster.makeDataDir call will setup the configuration for a > > MsgpackFormatConfig. I'll update my test cases to use it. > > > > However there is another question related to this. If I don't modify the > > boostrap-storage-plugins.json then when I launch the drill-embeded it > does > > not know of the MsgpackFormatConfig. I need to use the web console to > edit > > the storage plugins. > > > > Is there a better way? > > Thanks > > Jean-Claude > > > > > > On Wed, Sep 26, 2018 at 1:17 PM, Arina Yelchiyeva < > > arina.yelchiy...@gmail.com> wrote: > > > > > Taking into the account that your code is in the contrib module, > > modifying > > > boostrap-storage-plugins.json does not make any sense. > > > If you need to add your own format in unit tests, as Vitalii pointed > out, > > > TestCsv is a good example for this. > > > > > > Kind regards, > > > Arina > > > > > > On Wed, Sep 26, 2018 at 7:07 PM Jean-Claude Cote <jcc...@gmail.com> > > wrote: > > > > > > > I found the cause of the problem I had. It was not due to the fact > that > > > the > > > > classloader did not find the resource. It is that my new FormatPlugin > > was > > > > not registered into the > > > > > drill\exec\java-exec\src\main\resources\bootstrap-storage-plugins.json > > > > > > > > cp: { > > > > type: "file", > > > > connection: "classpath:///", > > > > formats: { > > > > "msgpack" : { > > > > type: "msgpack", > > > > extensions: [ "mp" ] > > > > }, > > > > > > > > So the resource was rejected. I've added this entry to the > > > > boostrap-storage-plugins.json and now it works. > > > > > > > > Thanks for all your help. > > > > Jean-Claude > > > > > > > > > > > > > > > > On Wed, Sep 26, 2018 at 7:18 AM, Arina Yelchiyeva < > > > > arina.yelchiy...@gmail.com> wrote: > > > > > > > > > This can also help: > > > > > > > > > > 1. create TEST resources folder where you'll copy your data. > > > > > 2. use dirTestWatcher to copy data to the root / tmp / custom > > > directory. > > > > > 3. query the data: > > > > > > > > > > @Test > > > > > public void t() { > > > > > > dirTestWatcher.copyResourceToRoot(Paths.get("complex_1.parquet")); > > > > > queryBuilder().sql("select * from dfs.`root`.`complex_1.parquet` > > > > > ").printCsv(); > > > > > } > > > > > > > > > > > > > > > On Wed, Sep 26, 2018 at 12:37 PM Vitalii Diravka < > vita...@apache.org > > > > > > > > wrote: > > > > > > > > > > > Hi Jean-Claude > > > > > > > > > > > > BaseTestQuery is deprecated. Please use ClusterTest instead. > > > > > > See TestCsv.java for example. > > > > > > > > > > > > You can find more info about Drill Cluster-Fixture-Framework > here: > > > > > > > > https://github.com/paul-rogers/drill/wiki/Cluster-Fixture-Framework > > > > > > > > > > > > On Wed, Sep 26, 2018 at 12:00 AM Jean-Claude Cote < > > jcc...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > I have writing a msgpack storage plugin from drill. > > > > > > > https://github.com/jcmcote/drill/tree/master/contrib/ > > > storage-msgpack > > > > > > > > > > > > > > I'm now trying to write test cases like > > > > > > > > > > > > > > testBuilder() > > > > > > > .sqlQuery("select * from cp.`msgpack/testBasic.mp`") > > > > > > > .ordered() > > > > > > > .baselineColumns("a").baselineValues("1"). > > > baselineValues("1") > > > > > > > .baselineColumns("b").baselineValues("2"). > > > baselineValues("2") > > > > > > > .build().run(); > > > > > > > > > > > > > > However when I run the test case it says it cannot find the > > > > > > > msgpack/testBasic.mp file. However it is in my > src/test/resources > > > > > folder. > > > > > > > > > > > > > > Should this work? I'm I going at it the right way? > > > > > > > Thanks > > > > > > > jc > > > > > > > > > > > > > > > > > > > > > > > > > > > >
