The proposed feature is to be able to pass options through SQL like you would when using the DataFrameReader API, so it would work for all sources that support read options. Read options are part of the DSv2 API, there just isn’t a way to pass options when using SQL. The PR also has a non-Iceberg example, which is being able to customize some JDBC source behaviors per query (e.g., fetchSize), rather than globally in the table’s options.
The proposed syntax is odd, but I think that's an artifact of Spark introducing read options that aren't a normal part of SQL. Seems reasonable to me to pass them through a hint. On Mon, Nov 15, 2021 at 2:18 AM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Interesting. > > What is this going to add on top of support for Apache Iceberg > <https://www.dremio.com/data-lake/apache-iceberg/>. Will it be in line > with support for Hive ACID tables or Delta Lake? > > HTH > > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Mon, 15 Nov 2021 at 01:56, Zhun Wang <wangzhun6...@gmail.com> wrote: > >> Hi dev, >> >> We are discussing Support Dynamic Table Options for Spark SQL ( >> https://github.com/apache/spark/pull/34072). It is currently not sure if >> the syntax makes sense, and would like to know if there is other feedback >> or opinion on this. >> >> I would appreciate any feedback on this. >> >> Thanks. >> > -- Ryan Blue Tabular
