andygrove commented on code in PR #57: URL: https://github.com/apache/datafusion-spark/pull/57#discussion_r4585465877
########## crates/spark/src/lib.rs: ########## @@ -11,22 +11,67 @@ #![deny(clippy::clone_on_ref_ptr)] #![cfg_attr(test, allow(clippy::needless_pass_by_value))] -//! Spark Expression packages for [DataFusion]. +//! [Apache Spark]-compatible expressions for [Apache DataFusion]. //! -//! This crate contains a collection of various Spark function packages for DataFusion, -//! implemented using the extension API. +//! DataFusion's built-in functions follow PostgreSQL / DuckDB conventions. When +//! the same function name exists in Apache Spark, the behavior often differs: +//! NULL handling, integer overflow, string indexing (1-based, negative +//! indices), rounding modes, date/time handling, error behavior in ANSI and +//! non-ANSI mode, and so on. This crate provides implementations of Spark +//! functions that follow Spark's semantics, so that engines built on DataFusion +//! (for example [Apache DataFusion Comet]) and applications migrating SQL from +//! Spark get the results Spark would produce. //! -//! [DataFusion]: https://crates.io/crates/datafusion +//! The crate depends only on the DataFusion building blocks (`datafusion-expr`, +//! `datafusion-common`, ...), not on the full `datafusion` crate, unless the +//! optional `core` feature is enabled. //! +//! # Getting started +//! +//! ```toml +//! [dependencies] +//! datafusion = "X" +//! # `core` enables `SessionStateBuilderSpark`, the easiest way to turn Spark mode on +//! datafusion-spark = { version = "X", features = ["core"] } +//! ``` //! //! # Available Function Packages -//! See the list of [modules](#modules) in this crate for available packages. +//! +//! Functions are grouped into packages by category (for example [`function::string`], +//! [`function::datetime`], [`function::math`], [`function::hash`]). See the list of +//! [modules](#modules) in this crate for the available packages. Each package Review Comment: Good idea, done in a0506c7. "Available packages" now links to [`function`]. ########## crates/spark/src/lib.rs: ########## @@ -11,22 +11,67 @@ #![deny(clippy::clone_on_ref_ptr)] #![cfg_attr(test, allow(clippy::needless_pass_by_value))] -//! Spark Expression packages for [DataFusion]. +//! [Apache Spark]-compatible expressions for [Apache DataFusion]. //! -//! This crate contains a collection of various Spark function packages for DataFusion, -//! implemented using the extension API. +//! DataFusion's built-in functions follow PostgreSQL / DuckDB conventions. When +//! the same function name exists in Apache Spark, the behavior often differs: +//! NULL handling, integer overflow, string indexing (1-based, negative +//! indices), rounding modes, date/time handling, error behavior in ANSI and +//! non-ANSI mode, and so on. This crate provides implementations of Spark +//! functions that follow Spark's semantics, so that engines built on DataFusion +//! (for example [Apache DataFusion Comet]) and applications migrating SQL from +//! Spark get the results Spark would produce. //! -//! [DataFusion]: https://crates.io/crates/datafusion +//! The crate depends only on the DataFusion building blocks (`datafusion-expr`, +//! `datafusion-common`, ...), not on the full `datafusion` crate, unless the +//! optional `core` feature is enabled. //! +//! # Getting started +//! +//! ```toml +//! [dependencies] +//! datafusion = "X" +//! # `core` enables `SessionStateBuilderSpark`, the easiest way to turn Spark mode on +//! datafusion-spark = { version = "X", features = ["core"] } +//! ``` //! //! # Available Function Packages -//! See the list of [modules](#modules) in this crate for available packages. +//! +//! Functions are grouped into packages by category (for example [`function::string`], +//! [`function::datetime`], [`function::math`], [`function::hash`]). See the list of +//! [modules](#modules) in this crate for the available packages. Each package +//! provides: +//! +//! * a `functions()` method returning all of its UDFs for registration +//! * an `expr_fn` module with helpers to build [`Expr`]s from Rust +//! * one accessor per function (for example `function::hash::sha2()`) returning the UDF +//! +//! Not every Spark function is implemented yet; coverage is growing. The +//! [`datafusion-spark` epic] tracks progress. +//! +//! # Semantics and caveats +//! +//! * **Registration overrides built-ins.** Spark functions are registered under +//! their Spark names. If DataFusion already has a function with that name, +//! the Spark one replaces it for the session. Register Spark functions after +//! DataFusion's defaults, and only when you want Spark behavior. +//! * **Input types.** Only types available in Spark are supported (no unsigned +//! integers, `Float16`, `Decimal64`, ...). +//! * **ANSI mode.** Functions whose Spark behavior depends on ANSI mode read +//! `datafusion.execution.enable_ansi_mode` from the session configuration. +//! * **Type inference.** DataFusion and Spark do not infer literal types in the +//! same way, so add explicit casts (`CAST(1 AS INT)`) when exact types matter. Review Comment: Added in a0506c7. "Semantics and caveats" has a new **SQL dialect** bullet saying that registering Spark functions does not change the parser, so `CAST(1 AS LONG)` needs the Spark dialect. `SessionStateBuilderSpark` now has a doctest that sets `sql_parser.dialect = Dialect::Spark`, passes the config to `with_config`, and runs `CAST(1 AS LONG)`. The existing `test_spark_dialect_with_spark_functions` unit test already checks that the generic dialect rejects `LONG`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
