[
https://issues.apache.org/jira/browse/SPARK-60641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60641:
-----------------------------------
Labels: pull-request-available (was: )
> Accept Byte and Short Row values for Float and Double schemas in Spark
> Connect Rust
> -----------------------------------------------------------------------------------
>
> Key: SPARK-60641
> URL: https://issues.apache.org/jira/browse/SPARK-60641
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> h2. Problem
> Existing local-input conversion gap identified while reviewing
> https://github.com/apache/spark-connect-rust/pull/109 (SPARK-60606); not
> introduced by that PR.
> In crates/spark-connect/src/session.rs, values_to_arrow's float_prim! accepts
> Value::Float, Double, Integer, Long, and Null, but not Byte or Short.
> Consequently a Rust Row containing a small integer (for example a value
> decoded from a tinyint/smallint column) fails with "Type mismatch in row
> data" when create_dataframe is given a Float or Double field. Wider
> Integer/Long inputs already succeed for the same declared targets. The
> removed coerce_value helper also lacked Byte/Short-to-float conversion, so
> retaining that pass would not fix the gap.
> h2. Minimal reproduction
> Construct Row::new(vec!["n".into()], vec![Value::Short(7)]) and pass it to
> SparkSession::create_dataframe with a Struct schema containing a nullable
> Float field named n. Local Arrow conversion currently returns a type mismatch
> before any remote execution. Repeat with Value::Byte(7) and a Double target;
> it fails for the same reason.
> h2. Expected behavior
> Handle Byte and Short consistently with the already-supported Integer and
> Long sources for Float and Double targets, using the encoder's existing
> numeric cast policy. This expectation concerns the native Rust Row conversion
> API; do not infer Python explicit-schema type-verification rules from the
> Rust behavior.
> h2. Acceptance criteria
> * Cover Byte and Short inputs for both Float and Double targets, including
> zero, negative values, type boundaries, and nulls. These integer ranges are
> exactly representable in both floating formats.
> * Exercise the full input-to-IPC path and verify output Arrow types and
> values.
> * Cover nested array, map-value, and struct leaves, which recurse through the
> same encoder.
> * Retain rejection of incompatible nonnumeric inputs and existing
> width/schema error ordering.
> * Include a live round trip of collected tinyint/smallint Rows into a new
> floating-schema DataFrame when a Spark server is available. Run Rust and
> relevant parity checks.
> Generated with OpenAI Codex.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
