+1 for adding OPTIONS hints. If we can do that in SQL extensions then that makes sense to me for the existing Spark versions that don't support it.
On Mon, Jun 26, 2023 at 11:18 AM Szehon Ho <szehon.apa...@gmail.com> wrote: > Hi, > > Yea that sounds good to me. > > Btw, that being said, I'm not opposed to configuring some of options in > the thread, especially write options, as sql conf either. (Not sure this > mechanism can support write conf without some changes to parser). And in > any case, it could be cascading: sql_dynamic_conf > sql_conf > table_conf. > Not sure what others think. > > Thanks, > Szehon > > On Sat, Jun 24, 2023 at 5:29 PM Manu Zhang <owenzhang1...@gmail.com> > wrote: > >> If the Spark community doesn’t accept this solution, how about adding it >> as an extension in Iceberg? I’m also wondering what people here think about >> it. >> >> Thanks for reviving the effort. >> Manu >> >> Szehon Ho <szehon.apa...@gmail.com>于2023年6月22日 周四00:45写道： >> >>> Hi, >>> >>> Yea, its definitely an issue. >>> >>> Fwiw, I was looking at reviving the old effort in Spark to pass in >>> configs dynamically in Spark SQL statement, which is probably the cleanest >>> solution. (https://github.com/apache/spark/pull/34072 was the old >>> effort, and I made https://github.com/apache/spark/pull/41683 based on >>> the suggestion from Spark community). Will keep the list posted. >>> >>> Thanks >>> Szehon >>> >>> On Fri, Jun 16, 2023 at 1:02 PM Wing Yew Poon >>> <wyp...@cloudera.com.invalid> wrote: >>> >>>> Hi, >>>> I recently put up a PR, https://github.com/apache/iceberg/pull/7790, >>>> to allow the write mode (copy-on-write/merge-on-read) to be specified in >>>> SQLConf. The use case is explained in the PR. >>>> Cheng Pan has an open PR, https://github.com/apache/iceberg/pull/7733, >>>> to allow locality to be specified in SQLConf. >>>> In the recent past, https://github.com/apache/iceberg/pull/6838/ was a >>>> PR to allow the write distribution mode to be specified in SQLConf. This >>>> was merged. >>>> Cheng Pan asks if there is any guidance on when we should allow configs >>>> to be specified in SQLConf. >>>> Thanks, >>>> Wing Yew >>>> >>>> ps. The above open PRs could use reviews by committers. >>>> >>>> -- Ryan Blue Tabular
