moomindani commented on PR #17765: URL: https://github.com/apache/iceberg/pull/17765#issuecomment-5591058614
@szehon-ho a nudge on this one — the question at the end of my 26 August comment is the only thing holding it, and I would rather have an answer either way than leave it sitting. To make it cheap to answer: the three routes to a long-lived SCD Type-2 table were (1) dual write, (2) SCD Type-2 as the base table, (3) derive from the changelog and store it. This PR is only the derivation step of (3) — it stores nothing itself. If you meant (1) or (2), those are different features and I do not claim this covers them. One correction: that comment pointed at `ef48e01a1` for the documentation, but I rebased the branch on 30 August and that SHA no longer exists. The docs are now in `40f83f48c`, under "Building a Long-Lived SCD Type-2 Table" in `docs/docs/spark-procedures.md`; it states the retention bound plainly and shows the incremental loop. And if your conclusion is that the retention window makes this not worth carrying inside the procedure, please say so and I will close the PR. Rebased onto current `main`, CI green (38/38). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
