DB Tsai created SPARK-60772:
-------------------------------
Summary: Accept inferred signed-int64 NumPy ndarray input in Spark
Connect Rust
Key: SPARK-60772
URL: https://issues.apache.org/jira/browse/SPARK-60772
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: DB Tsai
Fix For: connect-rust-4.3.0
Official Spark Connect accepts one- and two-dimensional NumPy ndarray inputs.
The current Rust binding and Arrow Table input PR118 recognize pandas or Table
inputs before casting the remainder to a list, so ordinary int64 ndarray input
remains unsupported.
Add a narrow path for nonempty ordinary exact native-endian signed-int64
ndarrays with omitted schema. Construct one Arrow array per physical column
without Python row materialization. Preserve official field names: value for
1D/one-column data, otherwise _1, _2, and so on. Keep physical integer types,
nullable metadata and every int64 value, including values beyond2**53. Do not
cast explicit logical schemas through the existing Table importer: official
ndarray input keeps its physical Arrow dtype and logical schema separate.
First verify the gap against the actual pinned installed official client and
failing Rust baseline, with independent captured LocalRelation IPC. Cover
layouts/strides/read-only arrays, integer extrema, width, source
lifetime/mutation, aliases/subclasses and unsupported shapes/dtypes. Explicit
StructType/DDL/AtomicType, dtype expansion, empty arrays and customized objects
are outside the initial supported scope; retain baseline behavior outside
accepted input. Existing list/Row/pandas/Table values, error order, hooks and
RPC behavior must remain unchanged. No pandas wrapper shortcut.
No ndarray case is timed by the existing353-case benchmark, and no speedup is
claimed over the baseline TypeError. Use separate frozen
baseline/candidate/official environments, focused correctness and first/warm
wall/CPU/RSS/RPC measurements, and unchanged existing-input guard harnesses. No
JVM/live Spark is needed for these client fixture checks.
This is distinct from PR118 Arrow Table input and PR121 pandas int64 input.
Source, GitHub PRs and Connect-version Apache issues were reviewed for overlap.
Scope is a narrow supported input addition, not complete NumPy parity.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
