kosiew commented on code in PR #57:
URL: https://github.com/apache/datafusion-spark/pull/57#discussion_r4582599818


##########
crates/spark/src/lib.rs:
##########
@@ -11,22 +11,67 @@
 #![deny(clippy::clone_on_ref_ptr)]
 #![cfg_attr(test, allow(clippy::needless_pass_by_value))]
 
-//! Spark Expression packages for [DataFusion].
+//! [Apache Spark]-compatible expressions for [Apache DataFusion].
 //!
-//! This crate contains a collection of various Spark function packages for 
DataFusion,
-//! implemented using the extension API.
+//! DataFusion's built-in functions follow PostgreSQL / DuckDB conventions. 
When
+//! the same function name exists in Apache Spark, the behavior often differs:
+//! NULL handling, integer overflow, string indexing (1-based, negative
+//! indices), rounding modes, date/time handling, error behavior in ANSI and
+//! non-ANSI mode, and so on. This crate provides implementations of Spark
+//! functions that follow Spark's semantics, so that engines built on 
DataFusion
+//! (for example [Apache DataFusion Comet]) and applications migrating SQL from
+//! Spark get the results Spark would produce.
 //!
-//! [DataFusion]: https://crates.io/crates/datafusion
+//! The crate depends only on the DataFusion building blocks 
(`datafusion-expr`,
+//! `datafusion-common`, ...), not on the full `datafusion` crate, unless the
+//! optional `core` feature is enabled.
 //!
+//! # Getting started
+//!
+//! ```toml
+//! [dependencies]
+//! datafusion = "X"
+//! # `core` enables `SessionStateBuilderSpark`, the easiest way to turn Spark 
mode on
+//! datafusion-spark = { version = "X", features = ["core"] }
+//! ```
 //!
 //! # Available Function Packages
-//! See the list of [modules](#modules) in this crate for available packages.
+//!
+//! Functions are grouped into packages by category (for example 
[`function::string`],
+//! [`function::datetime`], [`function::math`], [`function::hash`]). See the 
list of
+//! [modules](#modules) in this crate for the available packages. Each package

Review Comment:
   Consider linking "available packages" directly to [`function`] instead of 
`#modules`. The package descriptions are under `function/index.html`, so this 
would save readers a navigation step. The existing link works; this is only a 
discoverability suggestion.



##########
crates/spark/src/lib.rs:
##########
@@ -11,22 +11,67 @@
 #![deny(clippy::clone_on_ref_ptr)]
 #![cfg_attr(test, allow(clippy::needless_pass_by_value))]
 
-//! Spark Expression packages for [DataFusion].
+//! [Apache Spark]-compatible expressions for [Apache DataFusion].
 //!
-//! This crate contains a collection of various Spark function packages for 
DataFusion,
-//! implemented using the extension API.
+//! DataFusion's built-in functions follow PostgreSQL / DuckDB conventions. 
When
+//! the same function name exists in Apache Spark, the behavior often differs:
+//! NULL handling, integer overflow, string indexing (1-based, negative
+//! indices), rounding modes, date/time handling, error behavior in ANSI and
+//! non-ANSI mode, and so on. This crate provides implementations of Spark
+//! functions that follow Spark's semantics, so that engines built on 
DataFusion
+//! (for example [Apache DataFusion Comet]) and applications migrating SQL from
+//! Spark get the results Spark would produce.
 //!
-//! [DataFusion]: https://crates.io/crates/datafusion
+//! The crate depends only on the DataFusion building blocks 
(`datafusion-expr`,
+//! `datafusion-common`, ...), not on the full `datafusion` crate, unless the
+//! optional `core` feature is enabled.
 //!
+//! # Getting started
+//!
+//! ```toml
+//! [dependencies]
+//! datafusion = "X"
+//! # `core` enables `SessionStateBuilderSpark`, the easiest way to turn Spark 
mode on
+//! datafusion-spark = { version = "X", features = ["core"] }
+//! ```
 //!
 //! # Available Function Packages
-//! See the list of [modules](#modules) in this crate for available packages.
+//!
+//! Functions are grouped into packages by category (for example 
[`function::string`],
+//! [`function::datetime`], [`function::math`], [`function::hash`]). See the 
list of
+//! [modules](#modules) in this crate for the available packages. Each package
+//! provides:
+//!
+//! * a `functions()` method returning all of its UDFs for registration
+//! * an `expr_fn` module with helpers to build [`Expr`]s from Rust
+//! * one accessor per function (for example `function::hash::sha2()`) 
returning the UDF
+//!
+//! Not every Spark function is implemented yet; coverage is growing. The
+//! [`datafusion-spark` epic] tracks progress.
+//!
+//! # Semantics and caveats
+//!
+//! * **Registration overrides built-ins.** Spark functions are registered 
under
+//!   their Spark names. If DataFusion already has a function with that name,
+//!   the Spark one replaces it for the session. Register Spark functions after
+//!   DataFusion's defaults, and only when you want Spark behavior.
+//! * **Input types.** Only types available in Spark are supported (no unsigned
+//!   integers, `Float16`, `Decimal64`, ...).
+//! * **ANSI mode.** Functions whose Spark behavior depends on ANSI mode read
+//!   `datafusion.execution.enable_ansi_mode` from the session configuration.
+//! * **Type inference.** DataFusion and Spark do not infer literal types in 
the
+//!   same way, so add explicit casts (`CAST(1 AS INT)`) when exact types 
matter.

Review Comment:
   Consider clarifying that `with_spark_features()` does not select the Spark 
SQL parser dialect. Spark-specific syntax such as `CAST(1 AS LONG)` requires 
setting `config.options_mut().sql_parser.dialect = Dialect::Spark` and passing 
`config` to `SessionStateBuilder::with_config(config)`; the generic dialect 
rejects `LONG`. Not all supported queries require the Spark dialect.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to