yadavay-amzn commented on code in PR #56550:
URL: https://github.com/apache/spark/pull/56550#discussion_r3631994233
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala:
##########
@@ -359,17 +360,120 @@ object JdbcUtils extends Logging with SQLConfHelper {
internalRows.map(fromRow)
}
+ /**
+ * Builds an array of per-column sizing functions for InternalRow. Each
function returns the
+ * byte size contribution of that column (0 for nulls). Variable-length
types (String, Binary)
+ * use actual value size; fixed-width types use defaultSize.
+ *
+ * Shared by the read-path decode loop (columnSizers) and
measureInternalRowSize to avoid
+ * duplicated sizing logic.
+ */
+ private[jdbc] def buildInternalRowSizers(schema: StructType):
Array[InternalRow => Long] = {
+ schema.fields.zipWithIndex.map { case (field, idx) =>
+ field.dataType match {
+ case _: StringType => (row: InternalRow) =>
+ if (row.isNullAt(idx)) 0L else row.getUTF8String(idx).numBytes()
+ case BinaryType => (row: InternalRow) =>
+ if (row.isNullAt(idx)) 0L else row.getBinary(idx).length.toLong
+ case at: ArrayType => (row: InternalRow) =>
+ if (row.isNullAt(idx)) 0L
+ else {
+ val arr = row.getArray(idx)
+ at.elementType match {
+ case _: StringType =>
+ var s = 0L; var j = 0
+ while (j < arr.numElements()) {
+ if (!arr.isNullAt(j)) s += arr.getUTF8String(j).numBytes()
+ j += 1
+ }; s
+ case BinaryType =>
+ var s = 0L; var j = 0
+ while (j < arr.numElements()) {
+ if (!arr.isNullAt(j)) s += arr.getBinary(j).length
+ j += 1
+ }; s
+ case et => arr.numElements().toLong * et.defaultSize
Review Comment:
Thanks for catching this. Both sizers now iterate the array and skip null
elements (0 bytes each) and recurse into nested arrays, so `[1, null]` under
`ArrayType(IntegerType)` reports 4 and `[[1, 2], [3]]` reports 12. I applied
the same correction to the write-side `Seq` path, and added tests for null
elements, nested arrays, and empty/all-null arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
