RussellSpitzer edited a comment on issue #2934: URL: https://github.com/apache/iceberg/issues/2934#issuecomment-895293363
This is something that needs to be fixed in Iceberg, we haven't run into this yet but @manishmalhotrawork and I were talking about possible fixes over the weekend. I'm pretty sure the issue here is in predicate translation, we have a function sparkSql() which doesn't handle ByteBuffers even though those are valid predicates in evaluators. See https://github.com/apache/iceberg/blob/40e626a168a3137b42d023fb8f8f3835f3cd0a78/spark3/src/main/java/org/apache/iceberg/spark/Spark3Util.java#L718-L721 I think we just need to convert incoming byteBuffers to byteArrays here https://github.com/apache/iceberg/blob/40e626a168a3137b42d023fb8f8f3835f3cd0a78/spark3/src/main/java/org/apache/iceberg/spark/SparkFilters.java#L198-L209 So that the rest of our SparkUtil code will work with byte buffer literals generated by Spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
