ArijitSinghEDA opened a new issue, #11474:
URL: https://github.com/apache/iceberg/issues/11474
### Query engine
Spark
### Question
I am using Iceberg with PostgreSQL as catalog, MinIO as data storage and
using Spark for interacting with Iceberg. My application can take multiple
users working on the same table at the same time using Spark SQL. Now, when any
updates are made to this table (INSERT, UPDATE, and/or DELETE), a snapshot is
created, tracking these changes. Now, I wish to keep track of which user makes
what changes, and for that I wish to add the property userid to the snapshot
properties which are accessible through the summary column of the snapshot
table.
With INSERT command, I can convert it to a Spark SQL DataFrame, and run the
following to add userid to the snapshot properties
```
# Assuming df is the dataframe created for INSERT command
df.write.mode("append").option("snapshot-property.userid",
123).insertInto("iceberg.ns.tbl")
```
But, I cannot perform the same task while performing UPDATE or DELETE
command.
---
I tried to use branching as well, but when we try to merge the data back to
the main branch, the issue is that there should be an ID column for merging
data back, and if there are too many rows in the branched table, then merging
back will be a long and resource intensive task.
---
Is there any way I can pass the snapshot-property.userid as part of Spark
SQL query? I searched online, Iceberg's and Spark's documentations, as well as
ChatGPT, but nowhere was any solution regarding this.
If there is any better way than this, then also I am all ears.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
