JacbK commented on issue #14666: URL: https://github.com/apache/hudi/issues/14666#issuecomment-5064036727
I ran into this last year at Uber while working on DQ checks for batch-written Hudi tables. I was trying to figure out which write operation last touched each row from PySpark, but `_hoodie_operation` was null in our Spark snapshot reads. Looking at it again, I think `_hoodie_operation` is the wrong field for this anyway. What I need is the write operation from the commit metadata. Since each row already has `_hoodie_commit_time`, could `show_timeline` include the commit's `operationType` as `operation_type`? Then I could join on the instant time. This would just be the commit operation. An upsert commit would show `upsert` for every current row written by that commit, not whether each row was individually inserted or updated. Mostly want to make sure this is the right place to expose it before I put up a PR. I'm only looking at point 1 here, not the cross-table transaction metadata. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
