"storage": {
cp: {
type: "file",
connection: "classpath:///",
formats: {
"csv" : {
type: "text",
extensions: [ "csv" ],
delimiter: ","
}
}
enabled: true
}
}
"storage": {
dfs: {
type: "file",
connection: "file:///",
workspaces: {
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
},
"home": {
"location": "/Users/stefan",
"writable": true,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
},
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": null,
"allowAccessOutsideWorkspace": false
}
},
formats: {
"parquet": {
"type": "parquet"
},
"json": {
"type": "json",
"extensions": [
"json"
]
}
},
enabled: true
}
}
"storage": {
s3: {
type: "file",
connection: "s3a://<my-bucket-name>",
config: {
"fs.s3a.aws.credentials.provider":
"org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider",
"fs.s3a.endpoint": "s3.eu-central-1.amazonaws.com",
"fs.s3a.impl.disable.cache": "false"
},
workspaces: {
"root": {
"location": "/",
"writable": false,
"defaultInputFormat": "parquet",
"allowAccessOutsideWorkspace": false
}
},
"formats": {
"parquet": {
"type": "parquet"
}
},
enabled: true
}
}
On Mon, Jul 10, 2023 at 6:40 PM Charles Givre <[email protected]> wrote:
> Can you share your configs with any sensitive info redacted? The lists
> don't support images, so please just cut/paste the json.
> I had another idea...
> -- C
>
>
> > On Jul 10, 2023, at 12:28 PM, Stefan Ziegler <
> [email protected]> wrote:
> >
> > Yes, I think I'm following these instructions. And the file is not
> > completely ignored. It creates additional format definitions. Let's say I
> > white list some formats in my storage configuration and Drill adds more
> > formats (which I don't want). Is there another way to start a "vanilla"
> > Drill installation with my own configurations?
> >
> > Stefan
> >
> > On Mon, Jul 10, 2023 at 6:17 PM Charles Givre <[email protected]> wrote:
> >
> >> Hi Stefan,
> >> My apologies.. Ok.. so the issue is that the
> storage-plugins-override.conf
> >> is being ignored. I've never actually used this feature, so I wasn't
> >> familiar with it, but are you folllowing the instructions here [1] with
> >> respect to configuration and restarting Drill? My suggestion would be
> to
> >> remove all the plugins in the UI and only specify them in the .conf
> file.
> >> Drill has an order of precedence and I suspect what is happening is that
> >> the UI versions have a higher priority than the .conf versions. Does
> that
> >> make sense?
> >>
> >> -- C
> >>
> >> [1]:
> >>
> https://drill.apache.org/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-overrideconf-file
> >>
> >>
> >>
> >>> On Jul 10, 2023, at 12:06 PM, Stefan Ziegler <
> >> [email protected]> wrote:
> >>>
> >>> Hi Charles
> >>>
> >>> I use a "storage-plugins-override.conf" file. My attempt is to have the
> >>> configuration for my storages in a single file and Drill can pick up
> the
> >>> configuration on startup. I put "storage-plugins-override.conf" in the
> >> conf
> >>> directory and Drill creates the storages on startup but (and that is my
> >>> problem) also creates all formats for every storage defined in my
> config
> >>> file. E.g. I have a (local) file type storage and I define two formats
> >>> (parquet and json) in it. Drill does not respect my restriction to two
> >>> formats in the config file but creates all formats known to Drill (like
> >>> iceberg, xml etc.).
> >>>
> >>> regards
> >>> Stefan
> >>>
> >>> On Mon, Jul 10, 2023 at 5:30 PM Charles Givre <[email protected]>
> wrote:
> >>>
> >>>> HI Stefan,
> >>>> Thanks for your interest in Drill. You have to define the format
> config
> >>>> for each storage plugin. Otherwise Drill doesn't know what extension
> to
> >>>> associate with what format plugin. Out of curiosity, why are you
> using
> >> the
> >>>> .conf files for this?
> >>>> -- C
> >>>>
> >>>>
> >>>>> On Jul 9, 2023, at 12:03 PM, Stefan Ziegler <
> >> [email protected]>
> >>>> wrote:
> >>>>>
> >>>>> Not defining a format seems to prevent the user from querying the
> >>>> specific
> >>>>> format. E.g. after deleting the xml format definition in the web gui,
> >> I'm
> >>>>> not able to query xml files anymore. So I guess my assumption was
> >> right.
> >>>>>
> >>>>> Stefan
> >>>>>
> >>>>> On Sun, Jul 9, 2023 at 5:41 PM Stefan Ziegler <
> >>>> [email protected]>
> >>>>> wrote:
> >>>>>
> >>>>>> Btw: I assumed that the list of formats act as a restriction.
> Probably
> >>>> I'm
> >>>>>> wrong.
> >>>>>>
> >>>>>> Stefan
> >>>>>>
> >>>>>> On Sun, Jul 9, 2023 at 5:27 PM Stefan Ziegler <
> >>>> [email protected]>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi
> >>>>>>>
> >>>>>>> I'm using storage-plugins-override.conf to configure the storage
> >>>> plugins
> >>>>>>> on startup. My storage configurations contain only one or two
> formats
> >>>>>>> (parquet, json, csv). Checking the storages in the web gui I
> noticed
> >>>> that
> >>>>>>> for all the storages all formats are enabled, e.g. msaccess,
> iceberg
> >>>> etc.
> >>>>>>>
> >>>>>>> Is this on purpose or did I do something wrong?
> >>>>>>>
> >>>>>>> Example configuration:
> >>>>>>>
> >>>>>>> "storage": {
> >>>>>>> dfs: {
> >>>>>>> type: "file",
> >>>>>>> connection: "file:///",
> >>>>>>> workspaces: {
> >>>>>>> "tmp": {
> >>>>>>> "location": "/tmp",
> >>>>>>> "writable": true,
> >>>>>>> "defaultInputFormat": null,
> >>>>>>> "allowAccessOutsideWorkspace": false
> >>>>>>> },
> >>>>>>> "root": {
> >>>>>>> "location": "/",
> >>>>>>> "writable": false,
> >>>>>>> "defaultInputFormat": null,
> >>>>>>> "allowAccessOutsideWorkspace": false
> >>>>>>> }
> >>>>>>> },
> >>>>>>> formats: {
> >>>>>>> "parquet": {
> >>>>>>> "type": "parquet"
> >>>>>>> },
> >>>>>>> "json": {
> >>>>>>> "type": "json",
> >>>>>>> "extensions": [
> >>>>>>> "json"
> >>>>>>> ]
> >>>>>>> }
> >>>>>>> },
> >>>>>>> enabled: true
> >>>>>>> }
> >>>>>>> }
> >>>>>>>
> >>>>>>> regards
> >>>>>>> Stefan
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
