"storage": { cp: { type: "file", connection: "classpath:///", formats: { "csv" : { type: "text", extensions: [ "csv" ], delimiter: "," } } enabled: true } } "storage": { dfs: { type: "file", connection: "file:///", workspaces: { "tmp": { "location": "/tmp", "writable": true, "defaultInputFormat": null, "allowAccessOutsideWorkspace": false }, "home": { "location": "/Users/stefan", "writable": true, "defaultInputFormat": null, "allowAccessOutsideWorkspace": false }, "root": { "location": "/", "writable": false, "defaultInputFormat": null, "allowAccessOutsideWorkspace": false } }, formats: { "parquet": { "type": "parquet" }, "json": { "type": "json", "extensions": [ "json" ] } }, enabled: true } } "storage": { s3: { type: "file", connection: "s3a://<my-bucket-name>", config: { "fs.s3a.aws.credentials.provider": "org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider", "fs.s3a.endpoint": "s3.eu-central-1.amazonaws.com", "fs.s3a.impl.disable.cache": "false" }, workspaces: { "root": { "location": "/", "writable": false, "defaultInputFormat": "parquet", "allowAccessOutsideWorkspace": false } }, "formats": { "parquet": { "type": "parquet" } }, enabled: true } }
On Mon, Jul 10, 2023 at 6:40 PM Charles Givre <cgi...@gmail.com> wrote: > Can you share your configs with any sensitive info redacted? The lists > don't support images, so please just cut/paste the json. > I had another idea... > -- C > > > > On Jul 10, 2023, at 12:28 PM, Stefan Ziegler < > stefan.ziegler...@gmail.com> wrote: > > > > Yes, I think I'm following these instructions. And the file is not > > completely ignored. It creates additional format definitions. Let's say I > > white list some formats in my storage configuration and Drill adds more > > formats (which I don't want). Is there another way to start a "vanilla" > > Drill installation with my own configurations? > > > > Stefan > > > > On Mon, Jul 10, 2023 at 6:17 PM Charles Givre <cgi...@gmail.com> wrote: > > > >> Hi Stefan, > >> My apologies.. Ok.. so the issue is that the > storage-plugins-override.conf > >> is being ignored. I've never actually used this feature, so I wasn't > >> familiar with it, but are you folllowing the instructions here [1] with > >> respect to configuration and restarting Drill? My suggestion would be > to > >> remove all the plugins in the UI and only specify them in the .conf > file. > >> Drill has an order of precedence and I suspect what is happening is that > >> the UI versions have a higher priority than the .conf versions. Does > that > >> make sense? > >> > >> -- C > >> > >> [1]: > >> > https://drill.apache.org/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-overrideconf-file > >> > >> > >> > >>> On Jul 10, 2023, at 12:06 PM, Stefan Ziegler < > >> stefan.ziegler...@gmail.com> wrote: > >>> > >>> Hi Charles > >>> > >>> I use a "storage-plugins-override.conf" file. My attempt is to have the > >>> configuration for my storages in a single file and Drill can pick up > the > >>> configuration on startup. I put "storage-plugins-override.conf" in the > >> conf > >>> directory and Drill creates the storages on startup but (and that is my > >>> problem) also creates all formats for every storage defined in my > config > >>> file. E.g. I have a (local) file type storage and I define two formats > >>> (parquet and json) in it. Drill does not respect my restriction to two > >>> formats in the config file but creates all formats known to Drill (like > >>> iceberg, xml etc.). > >>> > >>> regards > >>> Stefan > >>> > >>> On Mon, Jul 10, 2023 at 5:30 PM Charles Givre <cgi...@gmail.com> > wrote: > >>> > >>>> HI Stefan, > >>>> Thanks for your interest in Drill. You have to define the format > config > >>>> for each storage plugin. Otherwise Drill doesn't know what extension > to > >>>> associate with what format plugin. Out of curiosity, why are you > using > >> the > >>>> .conf files for this? > >>>> -- C > >>>> > >>>> > >>>>> On Jul 9, 2023, at 12:03 PM, Stefan Ziegler < > >> stefan.ziegler...@gmail.com> > >>>> wrote: > >>>>> > >>>>> Not defining a format seems to prevent the user from querying the > >>>> specific > >>>>> format. E.g. after deleting the xml format definition in the web gui, > >> I'm > >>>>> not able to query xml files anymore. So I guess my assumption was > >> right. > >>>>> > >>>>> Stefan > >>>>> > >>>>> On Sun, Jul 9, 2023 at 5:41 PM Stefan Ziegler < > >>>> stefan.ziegler...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Btw: I assumed that the list of formats act as a restriction. > Probably > >>>> I'm > >>>>>> wrong. > >>>>>> > >>>>>> Stefan > >>>>>> > >>>>>> On Sun, Jul 9, 2023 at 5:27 PM Stefan Ziegler < > >>>> stefan.ziegler...@gmail.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi > >>>>>>> > >>>>>>> I'm using storage-plugins-override.conf to configure the storage > >>>> plugins > >>>>>>> on startup. My storage configurations contain only one or two > formats > >>>>>>> (parquet, json, csv). Checking the storages in the web gui I > noticed > >>>> that > >>>>>>> for all the storages all formats are enabled, e.g. msaccess, > iceberg > >>>> etc. > >>>>>>> > >>>>>>> Is this on purpose or did I do something wrong? > >>>>>>> > >>>>>>> Example configuration: > >>>>>>> > >>>>>>> "storage": { > >>>>>>> dfs: { > >>>>>>> type: "file", > >>>>>>> connection: "file:///", > >>>>>>> workspaces: { > >>>>>>> "tmp": { > >>>>>>> "location": "/tmp", > >>>>>>> "writable": true, > >>>>>>> "defaultInputFormat": null, > >>>>>>> "allowAccessOutsideWorkspace": false > >>>>>>> }, > >>>>>>> "root": { > >>>>>>> "location": "/", > >>>>>>> "writable": false, > >>>>>>> "defaultInputFormat": null, > >>>>>>> "allowAccessOutsideWorkspace": false > >>>>>>> } > >>>>>>> }, > >>>>>>> formats: { > >>>>>>> "parquet": { > >>>>>>> "type": "parquet" > >>>>>>> }, > >>>>>>> "json": { > >>>>>>> "type": "json", > >>>>>>> "extensions": [ > >>>>>>> "json" > >>>>>>> ] > >>>>>>> } > >>>>>>> }, > >>>>>>> enabled: true > >>>>>>> } > >>>>>>> } > >>>>>>> > >>>>>>> regards > >>>>>>> Stefan > >>>>>>> > >>>>>> > >>>> > >>>> > >> > >> > >