cj-zhukov opened a new issue, #26932:
URL: https://github.com/apache/datafusion/issues/26932
### Is your feature request related to a problem or challenge?
`DataFrame::from_columns` is a convenient way to build a `DataFrame` from
Arrow arrays, but it creates a new default `SessionContext` inside the
method.
The returned `DataFrame` therefore cannot use anything registered on the
session the caller already has: UDFs, tables, configuration, memory limits,
object stores, and so on.
For example, a UDF registered on `ctx` is not visible to a `DataFrame`
created with `from_columns`:
```rust
let ctx = SessionContext::new();
ctx.register_udf(add_one);
let df = DataFrame::from_columns([("x", x)])?;
// Error during planning: Invalid function 'add_one'
df.parse_sql_expr("add_one(x)")?;
```
The same applies to `write_table("t")` for a table registered on `ctx`,
to memory limits set on the runtime, and to object stores registered for
`write_parquet`. It also makes the result of a `join` depend on operand
order: `ctx_df.join(from_columns_df, ..)` keeps the caller's session,
`from_columns_df.join(ctx_df, ..)` keeps the throwaway one.
### Describe the solution you'd like
Add `SessionContext::read_columns`, next to `read_batch` and
`read_batches`, so a `DataFrame` built from arrays runs in the caller's
session like every other `DataFrame`:
```rust
let ctx = SessionContext::new();
ctx.register_udf(add_one);
let df = ctx.read_columns([("x", x)])?;
// Resolves and runs using the UDF registered on `ctx`
df.parse_sql_expr("add_one(x)")?;
```
Keeping two methods that do the same job is confusing, so
`DataFrame::from_columns` becomes a thin wrapper over
`SessionContext::new().read_columns(..)`, is marked `#[deprecated]`, and
will be removed after the usual deprecation period.
The `dataframe!` macro keeps its current API and switches to
`read_columns` internally.
### Describe alternatives you've considered
- Add a `SessionContext` or `SessionState` parameter to
`DataFrame::from_columns`. This is a breaking change and leaves the
constructor on `DataFrame`, while all other ways to create a `DataFrame`
from a data source live on `SessionContext`.
- Keep `from_columns` as is and document the limitation. This leaves the
method with the shortest name as the one that silently ignores the
caller's session.
### Additional context
`DataFrame::from_columns` was added together with the `dataframe!` macro in
#16104. The `DataFrame::new` docs already point users to
`SessionContext::read_*` methods for creating a `DataFrame` from a data
source; and `read_columns` should follow that convention.
### Willingness to contribute
I plan to contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
