Spark Merge On Read Support

2021-11-11 Thread Sreeram Garlapati
Hello Iceberg devs! After going through the mail threads (especially "Spark version support strategy") and relevant PRs - it looks like - *Merge on Read* Support (ie., Spark writers writing equality deletes) will be available with *Iceberg **+ Spark 3.2*. Is this understanding correct!? Or is this

Re: Regarding the support of pluggable procedures in Iceberg

2021-11-11 Thread Ryan Blue
I think there's a bit of a misunderstanding here. You shouldn't need to extend Iceberg's SparkCatalog to plug in stored procedures. The Iceberg Spark extensions should support stored procedures exposed by any catalog plugin that implements `ProcedureCatalog` across the Spark versions where Iceberg

Re: Regarding the support of pluggable procedures in Iceberg

2021-11-11 Thread Ryan Murray
Hey Ryan, What is the timeline for ProcedureCatalog to be moved into Spark and will it be backported? I agree 100% that its the 'correct' way to go long term but currently Iceberg has a `static final Map`[1] of valid procedures and no way for users to customize that. I personally don't love a stat