Can you share your configs with any sensitive info redacted? The lists don't support images, so please just cut/paste the json. I had another idea... -- C
> On Jul 10, 2023, at 12:28 PM, Stefan Ziegler <[email protected]> > wrote: > > Yes, I think I'm following these instructions. And the file is not > completely ignored. It creates additional format definitions. Let's say I > white list some formats in my storage configuration and Drill adds more > formats (which I don't want). Is there another way to start a "vanilla" > Drill installation with my own configurations? > > Stefan > > On Mon, Jul 10, 2023 at 6:17 PM Charles Givre <[email protected]> wrote: > >> Hi Stefan, >> My apologies.. Ok.. so the issue is that the storage-plugins-override.conf >> is being ignored. I've never actually used this feature, so I wasn't >> familiar with it, but are you folllowing the instructions here [1] with >> respect to configuration and restarting Drill? My suggestion would be to >> remove all the plugins in the UI and only specify them in the .conf file. >> Drill has an order of precedence and I suspect what is happening is that >> the UI versions have a higher priority than the .conf versions. Does that >> make sense? >> >> -- C >> >> [1]: >> https://drill.apache.org/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-overrideconf-file >> >> >> >>> On Jul 10, 2023, at 12:06 PM, Stefan Ziegler < >> [email protected]> wrote: >>> >>> Hi Charles >>> >>> I use a "storage-plugins-override.conf" file. My attempt is to have the >>> configuration for my storages in a single file and Drill can pick up the >>> configuration on startup. I put "storage-plugins-override.conf" in the >> conf >>> directory and Drill creates the storages on startup but (and that is my >>> problem) also creates all formats for every storage defined in my config >>> file. E.g. I have a (local) file type storage and I define two formats >>> (parquet and json) in it. Drill does not respect my restriction to two >>> formats in the config file but creates all formats known to Drill (like >>> iceberg, xml etc.). >>> >>> regards >>> Stefan >>> >>> On Mon, Jul 10, 2023 at 5:30 PM Charles Givre <[email protected]> wrote: >>> >>>> HI Stefan, >>>> Thanks for your interest in Drill. You have to define the format config >>>> for each storage plugin. Otherwise Drill doesn't know what extension to >>>> associate with what format plugin. Out of curiosity, why are you using >> the >>>> .conf files for this? >>>> -- C >>>> >>>> >>>>> On Jul 9, 2023, at 12:03 PM, Stefan Ziegler < >> [email protected]> >>>> wrote: >>>>> >>>>> Not defining a format seems to prevent the user from querying the >>>> specific >>>>> format. E.g. after deleting the xml format definition in the web gui, >> I'm >>>>> not able to query xml files anymore. So I guess my assumption was >> right. >>>>> >>>>> Stefan >>>>> >>>>> On Sun, Jul 9, 2023 at 5:41 PM Stefan Ziegler < >>>> [email protected]> >>>>> wrote: >>>>> >>>>>> Btw: I assumed that the list of formats act as a restriction. Probably >>>> I'm >>>>>> wrong. >>>>>> >>>>>> Stefan >>>>>> >>>>>> On Sun, Jul 9, 2023 at 5:27 PM Stefan Ziegler < >>>> [email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi >>>>>>> >>>>>>> I'm using storage-plugins-override.conf to configure the storage >>>> plugins >>>>>>> on startup. My storage configurations contain only one or two formats >>>>>>> (parquet, json, csv). Checking the storages in the web gui I noticed >>>> that >>>>>>> for all the storages all formats are enabled, e.g. msaccess, iceberg >>>> etc. >>>>>>> >>>>>>> Is this on purpose or did I do something wrong? >>>>>>> >>>>>>> Example configuration: >>>>>>> >>>>>>> "storage": { >>>>>>> dfs: { >>>>>>> type: "file", >>>>>>> connection: "file:///", >>>>>>> workspaces: { >>>>>>> "tmp": { >>>>>>> "location": "/tmp", >>>>>>> "writable": true, >>>>>>> "defaultInputFormat": null, >>>>>>> "allowAccessOutsideWorkspace": false >>>>>>> }, >>>>>>> "root": { >>>>>>> "location": "/", >>>>>>> "writable": false, >>>>>>> "defaultInputFormat": null, >>>>>>> "allowAccessOutsideWorkspace": false >>>>>>> } >>>>>>> }, >>>>>>> formats: { >>>>>>> "parquet": { >>>>>>> "type": "parquet" >>>>>>> }, >>>>>>> "json": { >>>>>>> "type": "json", >>>>>>> "extensions": [ >>>>>>> "json" >>>>>>> ] >>>>>>> } >>>>>>> }, >>>>>>> enabled: true >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> regards >>>>>>> Stefan >>>>>>> >>>>>> >>>> >>>> >> >>
