shuturmurgh opened a new pull request, #19067: URL: https://github.com/apache/pinot/pull/19067
Spark's DataSourceV2 write path applies an UnsafeProjection immediately before invoking DataWriter.write(...), so every row reaching PinotDataWriter is an UnsafeRow whose array fields are UnsafeArrayData. The old ArrayType branch called record.getArray(idx).array.map(...), but UnsafeArrayData.array() throws UnsupportedOperationException, so every job projecting an ArrayType column failed on the first row and users had to flatten multi-value columns to comma-delimited strings in Spark SQL and rebuild them server-side via ingestionConfig.transformConfigs. Rewrite the ArrayType branch to use accessors valid on both UnsafeArrayData and GenericArrayData -- per-element iteration over the typed getXxx(i) methods -- and emit Object[] of boxed primitives (or String[] / byte[][]) because Pinot's segment-generation path requires that shape: PinotBufferedRecordReader.next(reuse) calls GenericRow.copy() which casts every array value to (Object[]), and the pinot-segment-local stats collectors all do the same. Returning a primitive int[] / long[] would ClassCastException at the first stats pass. Other fixes in the same write path: - Field-level isNullAt guards in internalRowToGenericRow so the scalar StringType branch no longer NPEs on a null UTF8String and primitive scalar branches no longer silently substitute 0 / false for a null. - Reject null elements within multi-value arrays with IllegalArgumentException naming the column and index, since Pinot's per-type stats collectors NPE on null elements anyway. - Guard the time-column tracking in write() with isNullAt so a null time value does not silently pull startTime to 0 and corrupt getSegmentName. - Switch the scalar StringType branch to getUTF8String(idx).toString to drop the dependency on Spark's test-only InternalRow.getString. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
