DB Tsai created SPARK-60641:
-------------------------------
Summary: Accept Byte and Short Row values for Float and Double
schemas in Spark Connect Rust
Key: SPARK-60641
URL: https://issues.apache.org/jira/browse/SPARK-60641
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
h2. Problem
Existing local-input conversion gap identified while reviewing
https://github.com/apache/spark-connect-rust/pull/109 (SPARK-60606); not
introduced by that PR.
In crates/spark-connect/src/session.rs, values_to_arrow's float_prim! accepts
Value::Float, Double, Integer, Long, and Null, but not Byte or Short.
Consequently a Rust Row containing a small integer (for example a value decoded
from a tinyint/smallint column) fails with "Type mismatch in row data" when
create_dataframe is given a Float or Double field. Wider Integer/Long inputs
already succeed for the same declared targets. The removed coerce_value helper
also lacked Byte/Short-to-float conversion, so retaining that pass would not
fix the gap.
h2. Minimal reproduction
Construct Row::new(vec!["n".into()], vec![Value::Short(7)]) and pass it to
SparkSession::create_dataframe with a Struct schema containing a nullable Float
field named n. Local Arrow conversion currently returns a type mismatch before
any remote execution. Repeat with Value::Byte(7) and a Double target; it fails
for the same reason.
h2. Expected behavior
Handle Byte and Short consistently with the already-supported Integer and Long
sources for Float and Double targets, using the encoder's existing numeric cast
policy. This expectation concerns the native Rust Row conversion API; do not
infer Python explicit-schema type-verification rules from the Rust behavior.
h2. Acceptance criteria
* Cover Byte and Short inputs for both Float and Double targets, including
zero, negative values, type boundaries, and nulls. These integer ranges are
exactly representable in both floating formats.
* Exercise the full input-to-IPC path and verify output Arrow types and values.
* Cover nested array, map-value, and struct leaves, which recurse through the
same encoder.
* Retain rejection of incompatible nonnumeric inputs and existing width/schema
error ordering.
* Include a live round trip of collected tinyint/smallint Rows into a new
floating-schema DataFrame when a Spark server is available. Run Rust and
relevant parity checks.
Generated with OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
