cloud-fan commented on code in PR #56550:
URL: https://github.com/apache/spark/pull/56550#discussion_r3630560839
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala:
##########
@@ -359,17 +360,120 @@ object JdbcUtils extends Logging with SQLConfHelper {
internalRows.map(fromRow)
}
+ /**
+ * Builds an array of per-column sizing functions for InternalRow. Each
function returns the
+ * byte size contribution of that column (0 for nulls). Variable-length
types (String, Binary)
+ * use actual value size; fixed-width types use defaultSize.
+ *
+ * Shared by the read-path decode loop (columnSizers) and
measureInternalRowSize to avoid
+ * duplicated sizing logic.
+ */
+ private[jdbc] def buildInternalRowSizers(schema: StructType):
Array[InternalRow => Long] = {
+ schema.fields.zipWithIndex.map { case (field, idx) =>
+ field.dataType match {
+ case _: StringType => (row: InternalRow) =>
+ if (row.isNullAt(idx)) 0L else row.getUTF8String(idx).numBytes()
+ case BinaryType => (row: InternalRow) =>
+ if (row.isNullAt(idx)) 0L else row.getBinary(idx).length.toLong
+ case at: ArrayType => (row: InternalRow) =>
+ if (row.isNullAt(idx)) 0L
+ else {
+ val arr = row.getArray(idx)
+ at.elementType match {
+ case _: StringType =>
+ var s = 0L; var j = 0
+ while (j < arr.numElements()) {
+ if (!arr.isNullAt(j)) s += arr.getUTF8String(j).numBytes()
+ j += 1
+ }; s
+ case BinaryType =>
+ var s = 0L; var j = 0
+ while (j < arr.numElements()) {
+ if (!arr.isNullAt(j)) s += arr.getBinary(j).length
+ j += 1
+ }; s
+ case et => arr.numElements().toLong * et.defaultSize
Review Comment:
Null array elements must contribute 0 to an actual decoded-size metric, but
this branch counts every slot at `defaultSize`; `[1, null]` under
`ArrayType(IntegerType)` reports 8 bytes instead of 4. Please iterate over
these arrays, skip nulls, and recursively size nested arrays. The write-side
`seq.length * defaultSize` branch needs the same correction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
