Hi Péter, thanks for bringing this up. I don't think using a tag to "lock" a table is a good idea. The doc calls out that this is necessary "Since Flink doesn’t provide an out of the box solution for downstream operators sending feedback to upstream operators" so this feels like using Iceberg metadata as a side-channel within a Flink application. That doesn't seem like a good idea to me.
Why not use a separate Kafka topic to send control messages for this purpose, like what is done in the Kafka Connect sink? I think that's a cleaner way to solve the problem if there is not going to be a way to fix it in Flink. Ryan On Wed, Jul 31, 2024 at 7:45 AM Péter Váry <peter.vary.apa...@gmail.com> wrote: > Hi Team, > > During the discussion around the Flink Table Maintenance [1], [2], I have > highlighted that one of the main decision points is the way we prevent > concurrent Maintenance Tasks from happening concurrently. > > At that time we did not find better solution than providing an interface > for locking, and provide a basic implementation which is based on Iceberg > tags [3]: > > *"For convenience an Iceberg tag based solution is provided, so no > external dependencies are needed. On lock creation a tag named > '__flink_maitenance' is created for the current snapshot, and on lock > removal this tag is removed. This solution is not ideal, as it creates > multiple new versions of the table metadata, and mixes infrastructural > metadata with user metadata. If this is not acceptable then other > implementations could be created which could use external components like > JDBC/Kafka/ZooKeeper."* > > > We are in the implementation phase, and we agreed with Steven to take one > final round with the community, to see if anyone has a better suggestion, > or we could proceed with the originally agreed one. > > So if you have a better idea, please share. > > Thanks, > Peter > > [1] - https://lists.apache.org/thread/qjf83v2xj6lxb9sr8z0v9p0979f8wsmf > [2] - https://lists.apache.org/thread/vjf8m5wg840o58yz4y3q35k2mfhbm49l > [3] - > https://docs.google.com/document/d/16g3vR18mVBy8jbFaLjf2JwAANuYOmIwr15yDDxovdnA/edit#heading=h.lt9eaimi6zyz > -- Ryan Blue Databricks
