Re: Supports Dynamic Table Options for Spark SQL

My biggest concern with the syntax in hints is that Spark SQL's options can change results (e.g., CSV's header options) whereas hints are generally not designed to affect the external results if I am not mistaken. This is counterintuitive. I left the comment in the PR but what's the real benefit

Re: Supports Dynamic Table Options for Spark SQL

Side note about time travel: There is a PR to add VERSION/TIMESTAMP AS OF syntax to Spark SQL. On Mon, Nov 15, 2021 at 2:23 PM Ryan Blue wrote: > I want to note that I wouldn't recommend time traveling this way by using > the hint for `snapshot-id`.

Re: Supports Dynamic Table Options for Spark SQL

I want to note that I wouldn't recommend time traveling this way by using the hint for `snapshot-id`. Instead, we want to add the standard SQL syntax for that in a separate PR. This is useful for other options that help a table scan perform better, like specifying the target split size. You're

Re: Jira components cleanup

Done. Now let's see if that generated 86 update emails! On Mon, Nov 15, 2021 at 11:03 AM Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > > https://issues.apache.org/jira/projects/SPARK?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page > > I think the "docs" component

Re: Supports Dynamic Table Options for Spark SQL

I am looking at the hint and it appears to me (I stand corrected), it is a single table hint as below: -- time travel SELECT * FROM t /*+ OPTIONS('snapshot-id'='10963874102873L') */ My assumption is that any view on this table will also benefit from this hint. This is not a hint to optimizer in

Jira components cleanup

https://issues.apache.org/jira/projects/SPARK?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page I think the "docs" component should be merged into "Documentation". Likewise, the "k8" component should be merged into "Kubernetes". I think anyone can technically update tags, but

Re: Supports Dynamic Table Options for Spark SQL

I think since we probably will end up using this same syntax on write, this makes a lot of sense. Unless there is another good way to express a similar concept during a write operation I think going forward with this would be ok. On Mon, Nov 15, 2021 at 10:44 AM Ryan Blue wrote: > The proposed

Re: Supports Dynamic Table Options for Spark SQL

The proposed feature is to be able to pass options through SQL like you would when using the DataFrameReader API, so it would work for all sources that support read options. Read options are part of the DSv2 API, there just isn’t a way to pass options when using SQL. The PR also has a non-Iceberg

Re: Supports Dynamic Table Options for Spark SQL

Interesting. What is this going to add on top of support for Apache Iceberg . Will it be in line with support for Hive ACID tables or Delta Lake? HTH view my Linkedin profile