Copilot commented on code in PR #156:
URL: https://github.com/apache/hbase-connectors/pull/156#discussion_r3470835472
##########
spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/DefaultSource.scala:
##########
@@ -391,11 +392,20 @@ case class HBaseRelation(
val pushDownFilterJava =
if (usePushDownColumnFilter && pushDownDynamicLogicExpression != null) {
+ val columnMappings =
+ requiredQualifierDefinitionList.map {
+ field =>
+ new PushdownMappedField {
+ override def colName(): String = field.colName
Review Comment:
`SparkSQLPushDownFilter` executes `DynamicLogicExpression`s that rely on
their `encoder` field (used by `CompareTrait`). After moving these expression
classes to the pushdown module, the expression tree built in
`transverseFilterTree` is no longer implicitly tied to `HBaseRelation.encoder`,
and nothing here sets the encoder before sending the filter to the region
server. This will lead to an NPE at runtime when `encoder.filter(...)` is
called.
Set the encoder on the root expression before constructing the filter (and
ensure `setEncoder` propagates through the tree—see related comment in
`DynamicLogicExpression.scala`).
##########
hbase-connectors-assembly/src/main/assembly/hbase-connectors-bin.xml:
##########
@@ -34,12 +34,11 @@
<useAllReactorProjects>true</useAllReactorProjects>
<excludes>
<exclude>org.apache.hbase.connectors.spark:hbase-spark-it</exclude>
- <!-- Not yet packaged when assembly runs (reactor lists spark4 after
this module).
- Exclude until dist layout includes Spark 4; -->
+ <!-- Spark 4 line: not part of the default binary layout yet. -->
<exclude>org.apache.hbase.connectors.spark:hbase-spark4</exclude>
- <!-- Keep the Scala binary suffix in sync with scala.binary.version in
spark4/pom.xml;
- hbase-spark-pushdown artifactId is version-suffixed. -->
-
<exclude>org.apache.hbase.connectors.spark:hbase-spark-pushdown_${scala.binary.version}</exclude>
+ <!-- Internal pushdown libraries (pulled transitively by hbase-spark /
hbase-spark4).
+ ArtifactIds are Scala-suffixed; assembly descriptors do not
interpolate ${scala.binary.version}. -->
+
<exclude>org.apache.hbase.connectors.spark:hbase-spark_${spark4.scala.binary.version}</exclude>
</excludes>
Review Comment:
This exclude appears to target a non-existent artifactId
(`hbase-spark_${spark4.scala.binary.version}`); the Spark 3 connector
artifactId is `hbase-spark` (no Scala suffix) and the Scala-suffixed internal
artifact here is `hbase-spark-pushdown_${...}`. As written, the exclude is a
no-op and the internal pushdown jars will be packaged into the binary assembly.
Exclude the actual pushdown artifacts for both Scala lines instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
