kotwal-itpro commented on issue #5524:
URL: https://github.com/apache/hop/issues/5524#issuecomment-6161332263
All four parts of the plan are merged into main now, thanks to everyone who
reviewed them:
1. #8792: the lake table transforms and catalog moved into a shared
`tech/lakehouse` plugin (landed with #8802)
2. #8802: Lake table input reads Iceberg tables on the Hop engine
3. #8819: Lake table output writes Iceberg tables on the Hop engine, one
snapshot per pipeline run
4. #8859: docs, the new names, and a local-engine integration test
Along the way, #8795 fixed #8794, where the Spark engine wrote Iceberg PATH
tables to the temp folder.
So Lake table input and output now run on both the Hop engine and the native
Spark engine, as @hansva suggested ("design once, run everywhere"). Merge and
maintenance are still Spark only.
I'd leave this open for now, since the issue also covers Delta Lake. I have
Delta Lake reads on the Hop engine working locally with Delta Kernel, for #3941
(deletion vectors, time travel, parallel copies, and tables written by Spark).
If that's welcome, I'd propose it on #3941 as its own PR. Otherwise this issue
could be closed once 2.20 is out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
