o-nikolas commented on PR #73301:
URL: https://github.com/apache/airflow/pull/73301#issuecomment-5805700660
Hey @ash,
> We should probably cross link back to
`providers/amazon/docs/operators/duckdb.rst` from in the duckdb provider docs
too.
Sure, done. As well as the other comments above, either code changes made or
a reason provided why not.
>
> I'm still not sure about the idea of the duckdb_aws, connection type, but
having no Aws specific query operator is a massive improvment, thank you!
> Can this (be made to) work:
>
> with a connection defined as "duckdb_conn"
>
> ```json-comments
> {
> "extensions": ["aws"], // Or "s3" as you like
> "database": "s3://bucket/db.duckdb",
> "aws_conn_id": "some_aws_conn", // Only needed if not aws_default or not
ambient worker creds.
> }
> ```
>
> yielding:
>
> ```python
> DuckDBExecuteQueryOperator(
> conn_id="duckdb_conn" ,
> sql="SELECT ...",
> )
> ```
Unfortunately, not really, the `database` portion of the connection goes to
duckDB which treats it as a local or in-memory database (so you'll end up with
a local db file named `s3:/bucket/db.duckdb`), remote databases must be
`ATTACH`ed. Also something has to issue `CREATE SECRET` to wire up the
credentials. Also, I think configuring it like that with an extension is
getting a little esoteric for users, we pivoted from an operator to be more
user friendly in the first place.
I agree with you about the connection (read my comment above
[here](https://github.com/apache/airflow/pull/73301#issuecomment-5786643255))
which is why I simply wanted a single/small operator to be the mechanism to
point duck to AWS-land for users that just need a convenient way to have AWS
creds wired up for them. GCS can add their own operator (instead of their own
connection), etc, etc.
The connection approach works, it has it's pros and cons, and I'm fine to
keep it but I think it's ultimately more tedious IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
