uros-b commented on code in PR #56864:
URL: https://github.com/apache/spark/pull/56864#discussion_r3740620886
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:
##########
@@ -1490,6 +1490,92 @@ object VariantArrayAppendExpressionBuilder extends
VariantArrayAppendExpressionB
// scalastyle:on line.size.limit
object TryVariantArrayAppendExpressionBuilder
extends VariantArrayAppendExpressionBuilderBase(false)
+case class VariantStripNulls(child: Expression, includeArrays: Expression)
+ extends RuntimeReplaceable
+ with ExpectsInputTypes
+ with BinaryLike[Expression]
+ with QueryErrorsBase {
+
+ override def left: Expression = child
+ override def right: Expression = includeArrays
+
+ override def inputTypes: Seq[AbstractDataType] = Seq(VariantType,
BooleanType)
+
+ override def checkInputDataTypes(): TypeCheckResult = {
+ val result = super.checkInputDataTypes()
+ if (result.isFailure) {
+ result
+ } else if (!includeArrays.foldable) {
Review Comment:
The value is passed through StaticInvoke and read per row at runtime, so a
non-foldable boolean column would work here with no code changes. Could you
just please confirm, is this restriction deliberate?
It's also inconsistent with the closest sibling: VariantSet.createIfMissing
is a plain QuaternaryExpression child with no foldability check and accepts
dynamic expressions, even though its PySpark docstring claims "Must be a
constant." So today the family documents the same constraint in two places and
enforces it in one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
