as-sher opened a new issue, #11534:
URL: https://github.com/apache/iceberg/issues/11534
### Query engine
Spark 3.5
databricks Runtime 14.3 and 15.3
### Question
While trying to run
`spark.sql("CALL iceberg_catalog.system.expire_snapshots(table =>
'iceberg_catalog.d11_stitch.rewards_third_party_impact_base_query', older_than
=> TIMESTAMP '2024-03-06 00:00:00.000')")`
to this Im getting
```
Py4JJavaError: An error occurred while calling o415.sql.
: java.lang.NoSuchMethodError:
org.apache.spark.sql.catalyst.trees.Origin.<init>(Lscala/Option;Lscala/Option;Lscala/Option;Lscala/Option;Lscala/Option;Lscala/Option;Lscala/Option;)V
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergParserUtils$.position(IcebergSqlExtensionsAstBuilder.scala:377)
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergParserUtils$.withOrigin(IcebergSqlExtensionsAstBuilder.scala:367)
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSqlExtensionsAstBuilder.visitSingleStatement(IcebergSqlExtensionsAstBuilder.scala:334)
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSqlExtensionsAstBuilder.visitSingleStatement(IcebergSqlExtensionsAstBuilder.scala:66)
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSqlExtensionsParser$SingleStatementContext.accept(IcebergSqlExtensionsParser.java:153)
at
org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.$anonfun$parsePlan$1(IcebergSparkSqlExtensionsParser.scala:124)
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parse(IcebergSparkSqlExtensionsParser.scala:178)
at
org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parsePlan(IcebergSparkSqlExtensionsParser.scala:124)
at
com.databricks.sql.parser.DatabricksSqlParser.$anonfun$parsePlan$1(DatabricksSqlParser.scala:80)
at
com.databricks.sql.parser.DatabricksSqlParser.parse(DatabricksSqlParser.scala:101)
at
com.databricks.sql.parser.DatabricksSqlParser.parsePlan(DatabricksSqlParser.scala:77)
at
org.apache.spark.sql.SparkSession.$anonfun$sql$2(SparkSession.scala:892)
at
com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:94)
at
org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:452)
at
org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:891)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1180)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:890)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:924)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397)
at py4j.Gateway.invoke(Gateway.java:306)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at
py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199)
at py4j.ClientServerConnection.run(ClientServerConnection.java:119)
at java.lang.Thread.run(Thread.java:750)
File <command-2355211326393001>, line 1
----> 1 spark.sql("CALL iceberg_catalog.system.expire_snapshots(table =>
'iceberg_catalog.d11_stitch.rewards_third_party_impact_base_query', older_than
=> TIMESTAMP '2024-03-06 00:00:00.000')")
File
/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py:326, in
get_return_value(answer, gateway_client, target_id, name)
324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
325 if answer[1] == REFERENCE_TYPE:
--> 326 raise Py4JJavaError(
327 "An error occurred while calling {0}{1}{2}.\n".
328 format(target_id, ".", name), value)
329 else:
330 raise Py4JError(
331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
332 format(target_id, ".", name, value))
```
I tried using databricks Runtime 14.3 and 15.3 but getting the same issue on
both the runtimes.
I was able to run this successfully on 14.1
Im adding the following jars
iceberg-spark-runtime-3.5_2.12-1.5.0.jar iceberg-aws-bundle-1.5.0.jar
Can anyone help here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
