HyukjinKwon commented on PR #53:
URL:
https://github.com/apache/spark-connect-rust/pull/53#issuecomment-5408104564
**Addressed review item #2 (docs snippets not compile-tested) rigorously.**
I extracted every ```rust block from the docs and compiled each against the
real `spark_connect` crate (complete programs as-is; fragments wrapped in a
scaffold). All **15 doc code bins now compile, 0 failures** (the macro-based
`udfs.md` snippets were verified against the actual `#[spark_wasm_udf]` API and
the runnable `examples/wasm-udf-inline`).
Systematic errors found and fixed (they mirrored PySpark rather than the
Rust API):
- `spark.range(0, N)` -> `spark.range(N)` (takes one arg).
- Transformations return `DataFrame`, not `Result` -> removed stray `?`
after `select`/`filter`/`with_column`/`agg`.
- Actions (`count`/`show`/`collect`/`sql`) return `Result` -> keep `?`.
- Literals: `functions::lit` takes a `Column`; value literals must use the
crate-root `lit(i64)` / `lit_string(&str)` / `lit_double` / `lit_boolean`.
- Comparisons are methods (`.eq/.ne/.gt/.lt/.ge/.le`), not `==`/`>`;
arithmetic is `Column op Column` (scalars wrapped in `lit(...)`).
- Aggregation: `group_by(...).agg(vec![
f::sum(col).alias(..).expression().clone() ])`.
- I/O: `reader.load(Some(path))` returns `DataFrame`;
`writer.save(Some(path))?` returns `Result`.
- Config: `SparkSessionBuilder` has no `.config()` -> use
`spark.conf().set(k, v)?`.
- Streaming: import `spark_connect::streaming::Trigger`;
`reader.schema("<ddl>")`.
- Types: use `.cast_str("timestamp")` for a type-name string; `StructField {
name, data_type, nullable, metadata }` / `DataType::Map { key_type, value_type,
value_contains_null }` confirmed against the crate.
One crate-side note surfaced while doing this:
`spark_connect::functions::coalesce()` is currently a **0-argument stub** (it
can't take columns), so the docs now express null-fallback with
`when(...).otherwise(...)` instead. That looks like a separate crate gap worth
its own follow-up.
Pushed to `docs-site` (`798147e`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
