kosmldev opened a new issue, #11951:
URL: https://github.com/apache/iceberg/issues/11951
### Query engine
Spark 3.5, Hive Metastore
### Question
Functions CALL doesnt work, also doesnt work creating TAG and branches
my code:
`
conf = SparkConf()
conf.set('spark.hadoop.fs.s3a.access.key', s3_secret['aws_access_key_id'])
conf.set('spark.hadoop.fs.s3a.secret.key',
s3_secret['aws_secret_access_key'])
conf.set('spark.hadoop.fs.s3a.endpoint', s3_secret['endpoint_url'])
# Hive Metastore
conf.set(f'spark.hadoop.{hms_bucket}.access.key',
hms_secret['aws_access_key_id'])
conf.set(f'spark.hadoop.{hms_bucket}.secret.key',
hms_secret['aws_secret_access_key'])
conf.set('spark.sql.extensions','org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions')
conf.set("spark.sql.catalog.metastore_db",
"org.apache.iceberg.spark.SparkCatalog")
conf.set("spark.sql.iceberg.check-ordering", "false")
conf.set("spark.sql.catalog.metastore_db.type", "hive")
conf.set("spark.sql.catalog.metastore_db.uri", elf.hms_secret['hive_uris'])
conf.set("spark.jars.packages",
"org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.6.1,org.postgresql:postgresql:42.7.2")
spark = (
SparkSession.builder
.config(conf=conf)
.appName('alfor_2910')
.getOrCreate()
)
# table created like
test.spark.sql("""
CREATE TABLE metastore_db.spark_iceberg.test_funcions (
store_id int, item_id int, dt date, smth float
)
USING iceberg
PARTITIONED BY (
bucket(5, store_id),
bucket(5, item_id),
year(dt)
);
""")
# snapshots
test.spark.sql('select * from
metastore_db.spark_iceberg.test_funcions.snapshots').show()
#
+--------------------+-------------------+-------------------+---------+--------------------+--------------------+
# | committed_at| snapshot_id| parent_id|operation|
manifest_list| summary|
#
+--------------------+-------------------+-------------------+---------+--------------------+--------------------+
# |2025-01-13 03:49:...|3581078148379069374| null|
append|s3a://t-hive-meta...|{spark.app.id -> ...|
# |2025-01-13
03:49:...|1530700004686346841|3581078148379069374|overwrite|s3a://t-hive-meta...|{spark.app.id
-> ...|
# |2025-01-13
03:49:...|4471429915878262482|1530700004686346841|overwrite|s3a://t-hive-meta...|{spark.app.id
-> ...|
# |2025-01-13
03:49:...|5729427534040942866|4471429915878262482|overwrite|s3a://t-hive-meta...|{spark.app.id
-> ...|
#
+--------------------+-------------------+-------------------+---------+--------------------+--------------------+
# None of this doesn't work
# https://iceberg.apache.org/docs/latest/spark-procedures/
# https://iceberg.apache.org/docs/latest/spark-ddl/
# test.spark.sql("CALL
metastore_db.system.rollback_to_snapshot('spark_iceberg.test_funcions',
1530700004686346841);")
# test.spark.sql("CALL
metastore_db.system.expire_snapshots('spark_iceberg.test_funcions',
1530700004686346841);")
# test.spark.sql("CALL
metastore_db.system.rewrite_data_files('spark_iceberg.test_funcions');")
# test.spark.sql(f"ALTER TABLE metastore_db.spark_iceberg.test_funcions
CREATE TAG 'random tag' AS SNAPSHOT 1530700004686346841;")
`
Maybe i have wrong session config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
