Hi devs, I was about to give it a try, but it would relate to DSv2 so decide to initiate new thread before actual work. I also don't think this should be along with DSv2 discussion since the change would be minor.
While dealing with SPARK-24295 [1] and SPARK-26411 [2], I feel the needs of participating sink metadata into checkpoint directory, but unlike source which metadata directory is provided as subdirectory of checkpoint directory, sink doesn't receive its own metadata directory. For example, FileStreamSink creates metadata directory on output directory - though it is a bit intentional to share between queries - but sometimes we may want to make it coupled with query checkpoint. What do you think about passing metadata path to sink (we have only one for query) so that sink metadata can be coupled with query checkpoint? Thanks, Jungtaek Lim (HeartSaVioR) 1. https://issues.apache.org/jira/browse/SPARK-24295 2. https://issues.apache.org/jira/browse/SPARK-26411
