[jira] [Updated] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions
[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18185: --- Issue Type: Sub-task (was: Bug) Parent: SPARK-17861 > Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions > -- > > Key: SPARK-18185 > URL: https://issues.apache.org/jira/browse/SPARK-18185 > Project: Spark > Issue Type: Sub-task > Components: SQL >Reporter: Eric Liang > > As of current 2.1, INSERT OVERWRITE with dynamic partitions against a > Datasource table will overwrite the entire table instead of only the updated > partitions as in Hive. It also doesn't respect custom partition locations. > We should delete only the proper partitions, scan the metastore for affected > partitions with custom locations, and ensure that deletes/writes go to the > right locations for those as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions
[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18185: --- Target Version/s: 2.1.0 > Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions > -- > > Key: SPARK-18185 > URL: https://issues.apache.org/jira/browse/SPARK-18185 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Eric Liang > > As of current 2.1, INSERT OVERWRITE with dynamic partitions against a > Datasource table will overwrite the entire table instead of only the updated > partitions as in Hive. It also doesn't respect custom partition locations. > We should delete only the proper partitions, scan the metastore for affected > partitions with custom locations, and ensure that deletes/writes go to the > right locations for those as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions
[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18185: --- Description: As of current 2.1, INSERT OVERWRITE with dynamic partitions against a Datasource table will overwrite the entire table instead of only the updated partitions as in Hive. It also doesn't respect custom partition locations. We should delete only the proper partitions, scan the metastore for affected partitions with custom locations, and ensure that deletes/writes go to the right locations for those as well. was: As of current 2.1, INSERT OVERWRITE with dynamic partitions against a Datasource table will overwrite the entire table instead of only the updated partitions as in Hive. This is non-trivial to fix in 2.1, so we should throw an exception here. > Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions > -- > > Key: SPARK-18185 > URL: https://issues.apache.org/jira/browse/SPARK-18185 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Eric Liang > > As of current 2.1, INSERT OVERWRITE with dynamic partitions against a > Datasource table will overwrite the entire table instead of only the updated > partitions as in Hive. It also doesn't respect custom partition locations. > We should delete only the proper partitions, scan the metastore for affected > partitions with custom locations, and ensure that deletes/writes go to the > right locations for those as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions
[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18185: --- Summary: Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions (was: Should disallow INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions) > Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions > -- > > Key: SPARK-18185 > URL: https://issues.apache.org/jira/browse/SPARK-18185 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: Eric Liang > > As of current 2.1, INSERT OVERWRITE with dynamic partitions against a > Datasource table will overwrite the entire table instead of only the updated > partitions as in Hive. > This is non-trivial to fix in 2.1, so we should throw an exception here. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
