HI Stefan, What's in the config in the UI? Can you also please clarify what queries are running which indicate that your configs aren't working? Best, -- C
> On Jul 10, 2023, at 1:11 PM, Stefan Ziegler <[email protected]> > wrote: > > "storage": { > cp: { > type: "file", > connection: "classpath:///", > formats: { > "csv" : { > type: "text", > extensions: [ "csv" ], > delimiter: "," > } > } > enabled: true > } > } > "storage": { > dfs: { > type: "file", > connection: "file:///", > workspaces: { > "tmp": { > "location": "/tmp", > "writable": true, > "defaultInputFormat": null, > "allowAccessOutsideWorkspace": false > }, > "home": { > "location": "/Users/stefan", > "writable": true, > "defaultInputFormat": null, > "allowAccessOutsideWorkspace": false > }, > "root": { > "location": "/", > "writable": false, > "defaultInputFormat": null, > "allowAccessOutsideWorkspace": false > } > }, > formats: { > "parquet": { > "type": "parquet" > }, > "json": { > "type": "json", > "extensions": [ > "json" > ] > } > }, > enabled: true > } > } > "storage": { > s3: { > type: "file", > connection: "s3a://<my-bucket-name>", > config: { > "fs.s3a.aws.credentials.provider": > "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider", > "fs.s3a.endpoint": "s3.eu-central-1.amazonaws.com", > "fs.s3a.impl.disable.cache": "false" > }, > workspaces: { > "root": { > "location": "/", > "writable": false, > "defaultInputFormat": "parquet", > "allowAccessOutsideWorkspace": false > } > }, > "formats": { > "parquet": { > "type": "parquet" > } > }, > enabled: true > } > } > > > > > On Mon, Jul 10, 2023 at 6:40 PM Charles Givre <[email protected]> wrote: > >> Can you share your configs with any sensitive info redacted? The lists >> don't support images, so please just cut/paste the json. >> I had another idea... >> -- C >> >> >>> On Jul 10, 2023, at 12:28 PM, Stefan Ziegler < >> [email protected]> wrote: >>> >>> Yes, I think I'm following these instructions. And the file is not >>> completely ignored. It creates additional format definitions. Let's say I >>> white list some formats in my storage configuration and Drill adds more >>> formats (which I don't want). Is there another way to start a "vanilla" >>> Drill installation with my own configurations? >>> >>> Stefan >>> >>> On Mon, Jul 10, 2023 at 6:17 PM Charles Givre <[email protected]> wrote: >>> >>>> Hi Stefan, >>>> My apologies.. Ok.. so the issue is that the >> storage-plugins-override.conf >>>> is being ignored. I've never actually used this feature, so I wasn't >>>> familiar with it, but are you folllowing the instructions here [1] with >>>> respect to configuration and restarting Drill? My suggestion would be >> to >>>> remove all the plugins in the UI and only specify them in the .conf >> file. >>>> Drill has an order of precedence and I suspect what is happening is that >>>> the UI versions have a higher priority than the .conf versions. Does >> that >>>> make sense? >>>> >>>> -- C >>>> >>>> [1]: >>>> >> https://drill.apache.org/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-overrideconf-file >>>> >>>> >>>> >>>>> On Jul 10, 2023, at 12:06 PM, Stefan Ziegler < >>>> [email protected]> wrote: >>>>> >>>>> Hi Charles >>>>> >>>>> I use a "storage-plugins-override.conf" file. My attempt is to have the >>>>> configuration for my storages in a single file and Drill can pick up >> the >>>>> configuration on startup. I put "storage-plugins-override.conf" in the >>>> conf >>>>> directory and Drill creates the storages on startup but (and that is my >>>>> problem) also creates all formats for every storage defined in my >> config >>>>> file. E.g. I have a (local) file type storage and I define two formats >>>>> (parquet and json) in it. Drill does not respect my restriction to two >>>>> formats in the config file but creates all formats known to Drill (like >>>>> iceberg, xml etc.). >>>>> >>>>> regards >>>>> Stefan >>>>> >>>>> On Mon, Jul 10, 2023 at 5:30 PM Charles Givre <[email protected]> >> wrote: >>>>> >>>>>> HI Stefan, >>>>>> Thanks for your interest in Drill. You have to define the format >> config >>>>>> for each storage plugin. Otherwise Drill doesn't know what extension >> to >>>>>> associate with what format plugin. Out of curiosity, why are you >> using >>>> the >>>>>> .conf files for this? >>>>>> -- C >>>>>> >>>>>> >>>>>>> On Jul 9, 2023, at 12:03 PM, Stefan Ziegler < >>>> [email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Not defining a format seems to prevent the user from querying the >>>>>> specific >>>>>>> format. E.g. after deleting the xml format definition in the web gui, >>>> I'm >>>>>>> not able to query xml files anymore. So I guess my assumption was >>>> right. >>>>>>> >>>>>>> Stefan >>>>>>> >>>>>>> On Sun, Jul 9, 2023 at 5:41 PM Stefan Ziegler < >>>>>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Btw: I assumed that the list of formats act as a restriction. >> Probably >>>>>> I'm >>>>>>>> wrong. >>>>>>>> >>>>>>>> Stefan >>>>>>>> >>>>>>>> On Sun, Jul 9, 2023 at 5:27 PM Stefan Ziegler < >>>>>> [email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> I'm using storage-plugins-override.conf to configure the storage >>>>>> plugins >>>>>>>>> on startup. My storage configurations contain only one or two >> formats >>>>>>>>> (parquet, json, csv). Checking the storages in the web gui I >> noticed >>>>>> that >>>>>>>>> for all the storages all formats are enabled, e.g. msaccess, >> iceberg >>>>>> etc. >>>>>>>>> >>>>>>>>> Is this on purpose or did I do something wrong? >>>>>>>>> >>>>>>>>> Example configuration: >>>>>>>>> >>>>>>>>> "storage": { >>>>>>>>> dfs: { >>>>>>>>> type: "file", >>>>>>>>> connection: "file:///", >>>>>>>>> workspaces: { >>>>>>>>> "tmp": { >>>>>>>>> "location": "/tmp", >>>>>>>>> "writable": true, >>>>>>>>> "defaultInputFormat": null, >>>>>>>>> "allowAccessOutsideWorkspace": false >>>>>>>>> }, >>>>>>>>> "root": { >>>>>>>>> "location": "/", >>>>>>>>> "writable": false, >>>>>>>>> "defaultInputFormat": null, >>>>>>>>> "allowAccessOutsideWorkspace": false >>>>>>>>> } >>>>>>>>> }, >>>>>>>>> formats: { >>>>>>>>> "parquet": { >>>>>>>>> "type": "parquet" >>>>>>>>> }, >>>>>>>>> "json": { >>>>>>>>> "type": "json", >>>>>>>>> "extensions": [ >>>>>>>>> "json" >>>>>>>>> ] >>>>>>>>> } >>>>>>>>> }, >>>>>>>>> enabled: true >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> regards >>>>>>>>> Stefan >>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>> >>>> >> >>
