Re: schema changes of custom data source in persistent tables DataSourceV1

[fansparker]

Makes sense, Russell. I am trying to figure out if there is a way to enforce
metadata reload at "createRelation" if the provided schema in the new
sparkSession is different than the existing metadata schema. 



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: schema changes of custom data source in persistent tables DataSourceV1

[fansparker]

Thanks Russell.  This

  
shows that the "refreshTable" and "invalidateTable" could be used to reload
the metadata but they do not work in our case. I have tried to invoke the
"schema()" with the updated schema from the "buildScan()" as well. 

It will be helpful to have this feature enabled for DataSourceV1 as the
schema evolves, i will check if this is an change that can be made.

You mentioned that it works in DataSourceV2. Is there an implementation
sample for persistent tables DataSourceV2 that works with spark 2.4.4?
Thanks again.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: schema changes of custom data source in persistent tables DataSourceV1

[fansparker]

Does anybody know if there is a way to get the persisted table's schema
updated when the underlying custom data source schema is changed? Currently,
we have to drop and re-create the table. 



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

persistent tables in DataSource api V2

[fansparker]

1. In DataSource api V1, we were able to create persistent tables over custom
data sources using SQL DDL using "createRelation", "buildScan", "schema"
etc:. Is there a way to achieve this in DataSource api V2? 

2. In DataSource api V1, any schema changes in the underlying custom data
source is not reflected on the already persisted tables, even if the
"schema()" is re-invoked with the updated schema. Is there a way to get the
persisted table's schema updated? Thanks. 



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org