This is an automated email from the ASF dual-hosted git repository.
zhouyuan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 0f716e18e9 [GLUTEN-6887][VL] Daily Update Velox Version (2026_11_01)
(#13514)
0f716e18e9 is described below
commit 0f716e18e92d7ded83f1ca2fe5f78ada457b3f60
Author: Gluten Performance Bot
<[email protected]>
AuthorDate: Tue Nov 3 06:33:32 2026 +0000
[GLUTEN-6887][VL] Daily Update Velox Version (2026_11_01) (#13514)
* [GLUTEN-6887][VL] Daily Update Velox Version (dft-2026_11_01)
Upstream Velox's New Commits:
2cd961b5b by Rex(Hui) An, feat: Support UNKNOWN in Spark first and last
aggregates
8a97c9ed4 by GALI PREM SAGAR, fix(build): Pin macOS builds to Homebrew
OpenSSL 3
d8c3273e6 by Natasha Sehgal, refactor: Rename timestamp zone-label config
c853cce26 by Natasha Sehgal, feat(datetime)!: Unify AT TIME ZONE overloads
(#19632)
10d88f912 by Maria Basmanova, refactor(dwio)!: Move SortingWriter to the
Hive connector (#19930)
9bc5d6225 by rdtr, feat(spark): Add try_add, try_subtract and try_multiply
4404091da by Ke, perf(nimble): Defer loadNextChunk() on indexed skips
(#19899)
0bb70293a by shanhao-203, perf(nimble): Index Hybrid FlatMap writer keys
(#19924)
944677660 by Xiaoxuan, refactor(nimble): Remove BitRangeSplit encoding
(#19927)
a328dc713 by XiaoDu, fix(nimble): Reject corrupt bytes in encoding views
(#19911)
1e83dafc8 by Minni Mittal, feat(spark): Add right function
a618ba520 by XiaoDu, feat(nimble): Make hierarchical key encoding
configurable (#19904)
3c2d934df by shanhao-203, feat(nimble): Write and read Hybrid FlatMap
Default key lists (#19874)
c3976bc98 by Ke, perf(nimble): Skip chunks with no kept rows in sparse
reads (#19898)
3e6ec0221 by Reema, fix(cudf): Support Spark isnull expressions
71670679f by Rui Mo, test: Remove ORC decimal filter test
b0f990750 by Devavret Makkar, fix(cudf): Compare cast target types, not
kinds, in the AST evaluator
938be1cb7 by Rui Mo, misc: Update email
9e8ab84fa by oerling, feat(torchwave): Run and fuse the ads preproc graphs
(#19838)
6819f685d by Ke, fix(nimble): Keep row-size estimates at the end of a
stream (#19895)
4bb779018 by Maria Basmanova, test(exec): Add end-to-end vector search test
(#19915)
7b36d192e by shanhao-203, feat(nimble): Let the Hybrid FlatMap Default
group list its keys (#19873)
4eb621c7c by Maria Basmanova, feat(functions): Add vector_index_build
aggregate (#19886)
9d30a14c4 by XiaoDu, fix(nimble): Store one sequence per hierarchical key
level (#19902)
22f161ad3 by Pedro Pedreira, fix(vector): Copy struct rows whose children
are shorter
c957b8146 by Maria Basmanova, feat(exec): Add VectorSearchJoin (#19908)
35ec688bd by Richard Barnes, refactor(parquet): Remove unreachable returns
(#19810)
bcdab35c8 by RindsSchei225e, refactor(nimble): Move Hybrid FlatMap
configuration checks into common components (#19828)
fcdb1dec8 by Richard Barnes, refactor(velox): Remove unreachable returns
(#19811)
8ebb2ae28 by Maria Basmanova, fix(fuzzer): Give empty split groups a split
in JoinFuzzer
aeecbbab5 by macduan, fix: Reenable barrier tests
d79858715 by Xiaoxuan, Index RocksDB internal keys and decode entries in
read-ahead (#19878)
c7f56f6c3 by RindsSchei225e, fix(nimble): Fix NimbleEncodingRunner view
validation crash on nullable encodings (#19693)
b8e1609fe by peterenescu, fix(sparksql): Register unsuffixed
collect_set_merge_extract companion function (#19894)
45e4659d4 by peterenescu, refactor(sparksql): Split map functions into one
file each
310a576de by Krishna Pai, misc(rpc): Reconcile rate limiter floor with
fbcode (#19893)
f568730b9 by Maria Basmanova, refactor(exec): Check simple aggregates with
a concept (#19882)
1daffe2cd by XiaoDu, feat(nimble): Add PartitionedEliasFano encoding
(#19868)
6b05c9cf3 by Kevin Tang, fix(dwio): Share duplicate direct-read buffers
b6cb14e83 by Richard Barnes, refactor(cleanup): Remove unused constants in
Velox and storage code [unused-const-variable] (#19872)
37931b32a by Devavret Makkar, fix(fuzzer): Suspend drivers that grow the
aggregation fuzzer pool
d715f9e57 by Zhu, Lipeng, perf(functions): Force-inline Murmur3 hashInt64
to vectorize hash
f9a97a15b by mohsaka, fix(iceberg): Record null_value_count for NullType
Parquet columns
e583cd482 by GALI PREM SAGAR, fix(cudf): Give injected Iceberg TIMESTAMP
columns the reader's type
794626e2a by Maria Basmanova, feat(functions): Add vector_index_pack
aggregate (#19879)
73aeb82d4 by Wannuc-M, fix(rpc): Don't count a failed BATCH flush as a
success (#19875)
4861e3950 by RindsSchei225e, feat(nimble): Decode serde bool streams as
bits (#19800)
5df6f9b01 by tanjialiang, docs(nimble): Add a guide for adding required
features
1843f5237 by tanjialiang, feat(nimble): Load each stripe group once per
projection (#19826)
1bbf531de by zhichenxu-meta, feat(rpc): Cap paced probe gap and recover
with time (#19860)
07a9110ba by Pedro Pedreira, feat(functions): Support flat maps in
MakeRowFromMap (#19870)
5471326c1 by peterenescu, feat(sparksql): Implement native map_from_entries
(#19771)
75cce7758 by Amit Dutta, fix(serde): Revert size estimation and spill page
split
374848018 by Chandrashekhar Kumar Singh, feat(core): Copy registries
between query contexts (#19789)
43bd521ee by peterenescu, refactor(sparksql): Resolve map keys with LastWin
(#19770)
9268c48f6 by abhinavmuk04, fix(rpc): Honor each query's own rate-limiter
limits (#19794)
958e34532 by Bowei Xu, fix(from_json): Match Spark JSON semantics (#19725)
ded9ec4ee by tanjialiang, refactor(nimble): Retire stripe-group stream
checksums
99fb890dd by shanhao-203, feat(nimble): Make decompression verification
opt-in (#19829)
5beb2dfea by Maria Basmanova, fix(exec): Wake waiting consumers on buffer
terminate
e01ab9e8b by Maria Basmanova, fix(json): Fix json_parse buffer overflow on
constants
6b1f5a2b0 by dependabot[bot], build(ci): Bump astral-sh/setup-uv from 9.0.0
to 10.3.0
ba1dcdeed by Sahil Gandhi, feat(exec): Report each fixed-point iteration's
statistics (#19820)
655b5a9cc by Pramod Satya, fix(exec): Keep TopNRowNumber rank() correct
when a new row follows tied peers
04bd8fd23 by Xiaoxuan, fix(nimble): Keep null flat map values in scattered
reads
4e114bc08 by dependabot[bot], build(ci): Bump actions/cache/restore from
5.0.5 to 6.1.0
fa73c964c by dependabot[bot], build(ci): Bump docker/bake-action from 7.4.0
to 7.5.0
5cc68f4be by Maria Basmanova, fix(exec): Release splits when consumers
close early (#19831)
---
.../backendsapi/velox/VeloxSparkPlanExecApi.scala | 2 +-
.../gluten/expression/ExpressionTransformer.scala | 21 +++++++++++++
.../functions/JsonFunctionsValidateSuite.scala | 22 ++++++++++++++
ep/build-velox/src/get-velox.sh | 4 +--
.../apache/gluten/expression/ConverterUtils.scala | 34 ++++++++++++++++++----
.../gluten/utils/velox/VeloxTestSettings.scala | 4 +++
.../gluten/utils/velox/VeloxTestSettings.scala | 5 ++++
.../gluten/utils/velox/VeloxTestSettings.scala | 5 ++++
.../gluten/utils/velox/VeloxTestSettings.scala | 5 ++++
.../gluten/utils/velox/VeloxTestSettings.scala | 5 ++++
10 files changed, 98 insertions(+), 9 deletions(-)
diff --git
a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala
b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala
index 3a7497482f..0377d23c72 100644
---
a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala
+++
b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala
@@ -1276,7 +1276,7 @@ class VeloxSparkPlanExecApi extends SparkPlanExecApi with
Logging {
ExpressionNames.FROM_JSON,
FromJsonRestrictions.NOT_SUPPORT_COLUMN_CORRUPT_RECORD)
}
- GenericExpressionTransformer(substraitExprName, children, expr)
+ VeloxFromJsonTransformer(substraitExprName, children, expr)
}
/** Generate an expression transformer to transform StructsToJson to
Substrait. */
diff --git
a/backends-velox/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala
b/backends-velox/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala
index 7567a35663..cd51205c8d 100644
---
a/backends-velox/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala
+++
b/backends-velox/src/main/scala/org/apache/gluten/expression/ExpressionTransformer.scala
@@ -124,3 +124,24 @@ case class ToUnixTimestampTransformer(
}
}
}
+
+/**
+ * Transformer for Spark's `from_json`. Velox matches JSON keys against the
field names of the
+ * result type exactly (as Spark does), so the struct field names must not be
lowercased.
+ */
+case class VeloxFromJsonTransformer(
+ substraitExprName: String,
+ children: Seq[ExpressionTransformer],
+ original: Expression)
+ extends ExpressionTransformer {
+ override def doTransform(context: SubstraitContext): ExpressionNode = {
+ val funcName: String =
+ ConverterUtils.makeFuncName(substraitExprName,
original.children.map(_.dataType))
+ val functionId = context.registerFunction(funcName)
+ val childNodes = new JArrayList[ExpressionNode]()
+ children.foreach(c => childNodes.add(c.doTransform(context)))
+ val typeNode =
+ ConverterUtils.getTypeNode(dataType, nullable, preserveStructFieldNames
= true)
+ ExpressionBuilder.makeScalarFunction(functionId, childNodes, typeNode)
+ }
+}
diff --git
a/backends-velox/src/test/scala/org/apache/gluten/functions/JsonFunctionsValidateSuite.scala
b/backends-velox/src/test/scala/org/apache/gluten/functions/JsonFunctionsValidateSuite.scala
index d7566f5e93..785e5996b7 100644
---
a/backends-velox/src/test/scala/org/apache/gluten/functions/JsonFunctionsValidateSuite.scala
+++
b/backends-velox/src/test/scala/org/apache/gluten/functions/JsonFunctionsValidateSuite.scala
@@ -301,6 +301,28 @@ class JsonFunctionsValidateSuite extends
FunctionsValidateSuite {
}
}
+ test("from_json function with mixed-case field names") {
+ withTempPath {
+ path =>
+ Seq[String](
+ """{"Id":"10", "Nested":{"InnerA":1, "innerB":[{"DeepX":2}]}}""",
+ """{"id":"11", "Nested":{"innera":3, "innerB":[{"deepx":4}]}}""",
+ """{"Id":"12", "nested":{"InnerA":5}}"""
+ )
+ .toDF("txt")
+ .write
+ .parquet(path.getCanonicalPath)
+
+
spark.read.parquet(path.getCanonicalPath).createOrReplaceTempView("tbl")
+
+ runQueryAndCompare(
+ "select from_json(txt, 'Id STRING, " +
+ "Nested STRUCT<InnerA: INT, innerB: ARRAY<STRUCT<DeepX: INT>>>')
from tbl") {
+ checkGlutenPlan[ProjectExecTransformer]
+ }
+ }
+ }
+
test("from_json function CORRUPT_RECORD") {
withTempPath {
path =>
diff --git a/ep/build-velox/src/get-velox.sh b/ep/build-velox/src/get-velox.sh
index 920909c809..284f9b92a3 100755
--- a/ep/build-velox/src/get-velox.sh
+++ b/ep/build-velox/src/get-velox.sh
@@ -18,8 +18,8 @@ set -exu
CURRENT_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
VELOX_REPO=https://github.com/IBM/velox.git
-VELOX_BRANCH=dft-2026_10_30
-VELOX_ENHANCED_BRANCH=ibm-2026_10_30
+VELOX_BRANCH=dft-2026_11_01
+VELOX_ENHANCED_BRANCH=ibm-2026_11_01
VELOX_HOME=""
RUN_SETUP_SCRIPT=ON
ENABLE_ENHANCED_FEATURES=OFF
diff --git
a/gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala
b/gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala
index 3fc420ed55..2cd0fa9c06 100644
---
a/gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala
+++
b/gluten-substrait/src/main/scala/org/apache/gluten/expression/ConverterUtils.scala
@@ -222,7 +222,22 @@ object ConverterUtils extends Logging {
}
}
- def getTypeNode(datatype: DataType, nullable: Boolean): TypeNode = {
+ def getTypeNode(datatype: DataType, nullable: Boolean): TypeNode =
+ getTypeNode(datatype, nullable, preserveStructFieldNames = false)
+
+ /**
+ * Converts a Spark data type to a Substrait type node.
+ *
+ * @param preserveStructFieldNames
+ * If true, struct field names (including nested ones) are emitted as is,
regardless of
+ * `spark.sql.caseSensitive`. Needed by expressions whose semantics depend
on the exact field
+ * names of their result type, e.g. `from_json`, which matches JSON keys
against the schema
+ * field names case-sensitively.
+ */
+ def getTypeNode(
+ datatype: DataType,
+ nullable: Boolean,
+ preserveStructFieldNames: Boolean): TypeNode = {
datatype match {
case BooleanType =>
TypeBuilder.makeBoolean(nullable)
@@ -258,17 +273,24 @@ object ConverterUtils extends Logging {
case m: MapType =>
TypeBuilder.makeMap(
nullable,
- getTypeNode(m.keyType, nullable = false),
- getTypeNode(m.valueType, m.valueContainsNull))
+ getTypeNode(m.keyType, nullable = false, preserveStructFieldNames),
+ getTypeNode(m.valueType, m.valueContainsNull,
preserveStructFieldNames))
case a: ArrayType =>
- TypeBuilder.makeList(nullable, getTypeNode(a.elementType,
a.containsNull))
+ TypeBuilder.makeList(
+ nullable,
+ getTypeNode(a.elementType, a.containsNull, preserveStructFieldNames))
case s: StructType =>
val normalizeStructFieldName: String => String =
- if (BackendsApiManager.getSettings.structFieldToLowerCase()) {
+ if (
+ !preserveStructFieldNames &&
BackendsApiManager.getSettings.structFieldToLowerCase()
+ ) {
(name: String) => normalizeColName(name)
} else { (name: String) => name }
val (fieldNodes, fieldNames) = s.map {
- f => (getTypeNode(f.dataType, f.nullable),
normalizeStructFieldName(f.name))
+ f =>
+ (
+ getTypeNode(f.dataType, f.nullable, preserveStructFieldNames),
+ normalizeStructFieldName(f.name))
}.unzip
TypeBuilder.makeStruct(nullable, fieldNodes.asJava, fieldNames.asJava)
case _: NullType =>
diff --git
a/gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
b/gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
index cfb895a79b..104461c2a3 100644
---
a/gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
+++
b/gluten-ut/spark34/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
@@ -169,6 +169,10 @@ class VeloxTestSettings extends BackendTestSettings {
// Exception class different.
.exclude("from_json - invalid data")
enableSuite[GlutenJsonFunctionsSuite]
+ // Velox nulls the whole nested struct on a conversion error instead of
returning
+ // partial results for it (facebookincubator/velox#19725).
+ .exclude("SPARK-33134: return partial results only for root JSON objects")
+ .exclude("SPARK-40646: return partial results for objects with values as
JSON arrays")
// Velox does not support single quotes in get_json_object function.
.exclude("function get_json_object - support single quotes")
enableSuite[GlutenLiteralExpressionSuite]
diff --git
a/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
b/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
index 08c584714a..c97b6adaa0 100644
---
a/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
+++
b/gluten-ut/spark35/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
@@ -173,6 +173,11 @@ class VeloxTestSettings extends BackendTestSettings {
// Exception class different.
.exclude("from_json - invalid data")
enableSuite[GlutenJsonFunctionsSuite]
+ // Velox nulls the whole nested struct on a conversion error instead of
returning
+ // partial results for it (facebookincubator/velox#19725).
+ .exclude("SPARK-33134: return partial results only for root JSON objects")
+ .exclude("SPARK-40646: return partial results for objects with values as
JSON arrays")
+ .exclude("SPARK-48863: parse object as an array with partial results
enabled")
// * in get_json_object expression not supported in velox
.exclude("SPARK-42782: Hive compatibility check for get_json_object")
// Velox does not support single quotes in get_json_object function.
diff --git
a/gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
b/gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
index 5693a36a62..50198c58ab 100644
---
a/gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
+++
b/gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
@@ -183,6 +183,11 @@ class VeloxTestSettings extends BackendTestSettings {
// Exception class different.
.exclude("from_json - invalid data")
enableSuite[GlutenJsonFunctionsSuite]
+ // Velox nulls the whole nested struct on a conversion error instead of
returning
+ // partial results for it (facebookincubator/velox#19725).
+ .exclude("SPARK-33134: return partial results only for root JSON objects")
+ .exclude("SPARK-40646: return partial results for objects with values as
JSON arrays")
+ .exclude("SPARK-48863: parse object as an array with partial results
enabled")
// * in get_json_object expression not supported in velox
.exclude("SPARK-42782: Hive compatibility check for get_json_object")
// Velox does not support single quotes in get_json_object function.
diff --git
a/gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
b/gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
index 781cfb2eee..d96a2b0bd8 100644
---
a/gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
+++
b/gluten-ut/spark41/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
@@ -189,6 +189,11 @@ class VeloxTestSettings extends BackendTestSettings {
// Exception class different.
.exclude("from_json - invalid data")
enableSuite[GlutenJsonFunctionsSuite]
+ // Velox nulls the whole nested struct on a conversion error instead of
returning
+ // partial results for it (facebookincubator/velox#19725).
+ .exclude("SPARK-33134: return partial results only for root JSON objects")
+ .exclude("SPARK-40646: return partial results for objects with values as
JSON arrays")
+ .exclude("SPARK-48863: parse object as an array with partial results
enabled")
// * in get_json_object expression not supported in velox
.exclude("SPARK-42782: Hive compatibility check for get_json_object")
// Velox does not support single quotes in get_json_object function.
diff --git
a/gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
b/gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
index b3ed0b3783..f39b80e634 100644
---
a/gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
+++
b/gluten-ut/spark42/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
@@ -214,6 +214,11 @@ class VeloxTestSettings extends BackendTestSettings {
// Exception class different.
.exclude("from_json - invalid data")
enableSuite[GlutenJsonFunctionsSuite]
+ // Velox nulls the whole nested struct on a conversion error instead of
returning
+ // partial results for it (facebookincubator/velox#19725).
+ .exclude("SPARK-33134: return partial results only for root JSON objects")
+ .exclude("SPARK-40646: return partial results for objects with values as
JSON arrays")
+ .exclude("SPARK-48863: parse object as an array with partial results
enabled")
// * in get_json_object expression not supported in velox
.exclude("SPARK-42782: Hive compatibility check for get_json_object")
// Velox does not support single quotes in get_json_object function.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
