HeartSaVioR edited a comment on pull request #33749: URL: https://github.com/apache/spark/pull/33749#issuecomment-900821635
I see the benefit of ability on overriding the table format version, but it also brings confusion as well. This is not the same as we have been doing from state format version. For state format version, we restrict modifying the value once it is set in any way (regardless default value or user defined value), so the value is unchanged in the lifetime of the query. Here we allow to set any value to override the config, which leads that multiple table format versions exist in SST files. This won't hurt if RocksDB version in Spark can understand all of these versions, but it is still less simpler. If we'd like to make this simpler as possible, it'd be better if we just disallow overriding the config. If we still want to open the chance to override based on the behavior that having multiple table format versions doesn't matter, let's prevent setting "lower" version than checkpoint to ensure there are no SST file having higher table format version than checkpoint config. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
